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Dr.  William  Raub 

Acting  Director 

National  Institutes  of  Health 

Shannon  Building 

9000  Rockville  Pike 

Bethesda,  Maryland  20892 

Dear  Dr.  Raub: 

I  am  pleased  to  transmit  to  you  the  final  report  of  the  ad 
hoc  consultant  panel  to  the  Advisory  Committee  to  the  Director, 
NIH,  on  "Strengthening  the  Scientific  Review  Procedures  of  the 
NIH  Intramural  Research  Program". 

The  panel  charge  was  to  examine  all  aspects  of  the  current 
NIH  intramural  scientific  review  procedures  and  to  make 
recommendations  for  strengthening  the  review  process.  As  part  of 
its  mandate,  the  panel  also  was  to  consider  the  recommendations 
of  the  National  Academy  of  Sciences,  Institute  of  Medicine 
(NAS/IOM)  for  modifying  the  intramural  review  process,  to 
determine  whether  the  NAS/IOM  recommendations  would  strengthen 
the  intramural  research  program,  and  if  so,  how  they  could  be 
implemented  most  effectively. 

The  panel  report  reviews  the  genesis  of  this  issue  and  the 
substantive  information  considered  by  the  panel,  including  the 
NAS/IOM  study,  testimony  on  the  intent  of  the  study  committee's 
recommendations,  the  regulations  that  define  the  current  review 
procedures,  and  the  application  of  those  regulations  to  several 
of  the  NIH  institutes.  The  report  also  examines  proposals  for 
improving  the  intramural  review  process  that  were  presented  by  a 
working  group  from  the  NIH  intramural  research  program.  Many  of 
these  proposals  were  believed  by  the  panel  to  be  quite 
meritorious  and  these  are  reflected  in  the  panel's  final 
recommendations.  Finally,  the  report  provides  the  deliberations 
of  the  panel,  and  its  summary  recommendations  for  strengthening 
the  scientific  review  procedures  of  the  NIH  intramural  research 
program . 

The  panel's  recommendations  are  intended  to  enhance  existing 
review  procedures  and  should  not  be  interpreted  as  a  minimum  set 
of  guidelines  in  and  of  themselves.  The  panel  found  the  existing 
procedures  to  be  on  the  whole  sound  and  sought  to  provide  its 
guidance  on  how  these  procedures  could  be  improved. 
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In  closing,  I  would  like  to  emphasize  a  belief  of  the  panel 
that  is  related  to  the  broader  issue  of  maintaining  the 
scientific  excellence  of  the  NIH  intramural  research  program. 
Although  rigorous  scientific  review  procedures  lead  to  greater 
research  accountability,  the  strength  of  the  NIH  intramural 
research  program  ultimately  lies  with  the  talent  of  the 
scientists  who  oversee  and  guide  the  enterprise.  The  panel  fully 
concurs  with  the  NAS/IOM  finding  that  the  responsibilities  of  the 
scientific  director  reguire  both  scientific  vision  and  well-honed 
management  skills  and  endorses  the  NAS/IOM  recommendation  that 
additional  compensation  should  be  available  to  recruit  and  retain 
scientists  of  the  highest  caliber. 

The  panel  adopted  the  report  unanimously  and  trusts  that  you 
will  find  the  recommendations  useful  in  strengthening  the  NIH 
intramural  research  program. 


Sincerely  yours 


Robert  L.  Hill,  Ph.D. 
Chairperson 

Consultant  Panel  of  the 
Advisory  Committee  to  the 
Director,  NIH 
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Background 

Intramural  research  at  the  National  Institutes  of  Health  (NIH)  has  been  reviewed 
by  committees  of  scientists  from  outside  NIH  since  1956,  when  the  first  commit¬ 
tees,  called  boards  of  scientific  counselors,  were  appointed  to  review  intramural 
research  in  the  individual  institutes.  These  boards  of  scientific  counselors  are 
public  advisory  committees  established  under  provisions  of  the  Federal  Advis¬ 
ory  Committee  Act  (Public  Law  92^163).  They  review,  discuss,  and  evaluate 
research  projects  and  programs  conducted  by  NIH  as  well  as  intramural  research 
programs  of  the  Alcohol,  Drug  Abuse,  and  Mental  Health  Administration 
(ADAMHA)  located  on  the  NIH  campus.  The  reviews  include  evaluation  of  the 
work  of  individual  scientists  as  well  as  of  the  quality  of  the  research  itself. 

During  the  development  of  the  fiscal  year  (FY)  1989  President's  Budget,  the 
Office  of  Management  and  Budget  (OMB)  requested  the  Department  of  Health 
and  Human  Services  (DHHS)  to  evaluate  strategies  to  strengthen  the  scientific 
excellence  of  the  NIH  intramural  laboratories  through  an  independent  study  by 
the  National  Academy  of  Sciences  and  the  Institute  of  Medicine  (NAS/IOM).  In 
December  1988,  NAS/IOM  issued  its  report,  A  Healthy  NIH  Intramural  Program: 
Structural  Change  or  Administrative  Remedies?  Although  the  study  concluded  that 
"the  intramural  program  has  made  and  continues  to  make  invaluable  contribu¬ 
tions  to  our  knowledge  and  understanding  of  basic  biological  processes  and  their 
dysfunction  in  disease,"  the  report  also  expressed  concerns  regarding  the  accoun¬ 
tability  and  independence  of  the  scientific  review  procedures  of  the  NIH  intra¬ 
mural  research  program.  Among  its  several  recommendations,  the  NAS/IOM 
study  committee  offered  two  that  specifically  addressed  the  NIH  scientific 
review  procedures: 

"A  panel  chaired  by  a  member  of  the  NIH  Director's  Advisory  Commit¬ 
tee  should  be  established  to  monitor  the  intramural  research  program 
review";  and 

"Each  of  the  scientific  directors  and  their  intramural  programs  should 
be  reviewed  as  a  whole  every  four  years  by  an  external  group.  The 
review  report  should  be  submitted  to  the  director  of  the  relevant  institute. 
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the  NTH  Deputy  Director  for  Intramural  Research,  the  Director  of  NIH, 
and  the  Director's  Advisory  Committee." 

In  response  to  these  two  recommendations,  the  Director  of  NIH  established  a 
Consultant  Panel  to  the  Advisory  Committee  to  the  Director  (DAC),  NIH,  to 
examine  all  aspects  of  the  current  NIH  intramural  scientific  review  procedures 
and  to  make  recommendations  for  strengthening  the  intramural  review  process. 

A  meeting  of  the  Panel  was  held  at  NIH  on  September  5, 1989.  In  advance  of  the 
meeting,  the  Panel  was  provided  with  briefing  materials  including  the 
NAS/IOM  report;  Chapter  3005  of  the  DHHS  manual,  which  establishes  the 
policy  and  procedures  for  outside  review  and  evaluation  of  intramural  research 
at  NIH;  a  working  paper  prepared  by  NIH  staff  that  suggested  changes  to  the 
current  review  procedures;  and  a  survey  and  analysis  of  intramural  evaluation 
prepared  by  the  NIH  Office  of  Intramural  Research. 

During  this  meeting,  the  Panel  heard  testimony  regarding  the  intent  of  the 
NAS/IOM  study  committee's  recommendations  from  Dr.  Robert  Petersdorf,  a 
member  of  the  original  NAS/IOM  study  committee.  The  Panel  also  heard 
presentations  from  senior  scientists  from  the  NIH  institutes,  including  institute 
directors,  and  the  scientific  directors  of  the  intramural  research  programs. 

Summary  of  Issues  Discussed 

In  the  course  of  its  deliberations,  the  Panel  addressed  the  two  study  committee 
recommendations  and  several  related  issues  raised  in  the  NAS/IOM  report.  It 
examined  the  procedures  for  selecting  reviewers  and  the  mechanisms  for  ensur¬ 
ing  that  there  is  an  appropriate  response  to  each  review.  The  Panel  was  con¬ 
cerned  about  negative  perceptions  of  the  review  procedures,  particularly  those 
that  called  into  question  the  accountability  and  independence  of  the  intramural 
review  process.  Among  the  issues  addressed  by  the  Panel  were  the  following: 

•  How  should  counselors  be  selected  and  what  is  the  optimum  composition, 
size,  and  length  of  service  of  the  boards  of  scientific  counselors? 

•  Is  there  a  need  for  a  minimum  set  of  codified  standards  that  would  apply 
across  all  the  institutes? 

•  What  kind  of  orientation  do  counselors  receive? 

•  How  should  the  review  be  structured,  to  whom  should  the  report  of  the 
review  be  delivered,  and  what  should  be  the  mechanisms  for  following 
through? 

•  What  is  the  most  appropriate  mechanism  for  responding  to  the  report  of  the 
review  by  the  boards  of  scientific  counselors? 

The  Panel  considered  the  perception  expressed  by  some  individuals  outside  NIH 
that  the  NIH  intramural  scientific  review  process  is  less  demanding  than  the 
extramural  research  peer  review  process.  Questions  regarding  the  rigor. 
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uniformity,  and  followthrough  of  the  review  procedures  were  also  discussed  by 
the  Panel. 

The  Panel  also  found  that  the  tradition  of  scientific  review  at  the  NIH  dates  back 
to  the  establishment  of  the  first  boards  of  scientific  counselors  in  1956,  and  that 
the  review  procedures  have  been  strengthened  periodically  through  the  publica¬ 
tion  of  revised  policy  and  procedures  contained  in  Chapter  3005. 

Finally,  the  consultants  felt  strongly  that  the  extramural  and  intramural  research 
communities  were  sufficiently  unique  to  require  different  approaches  to  scien¬ 
tific  evaluation.  Unlike  the  extramural  community's  emphasis  on  prospective 
peer  review  and  its  focus  on  potential,  the  NIH  intramural  scientific  review 
process  is  a  critique  of  past  performance  designed  to  elicit  counsel  regarding  cur¬ 
rent  and  future  activities. 

The  following  Panel  recommendations  for  strengthening  the  intramural  review 
procedures  represent  new  proposals  or  modifications  to  existing  requirements 
outlined  in  Chapter  3005. 

Summary  of  Recommendations 

Monitoring  of  the  Intramural  Research  Program 

1.  An  ad  hoc  Consultant  Panel  to  the  Advisory  Committee  to  the  Director,  NIH, 
should  be  convened  in  2  years,  and  periodically  thereafter,  to  review  progress 
on  the  following  recommendations  and  changes  in  the  intramural  scientific 
review  process. 

Evaluation  of  the  Scientific  Director 

2.  The  scientific  and  administrative  performance  of  the  scientific  directors 
should  be  evaluated  every  4  years  by  the  boards  of  scientific  counselors.  The 
chairperson  of  the  board  of  scientific  counselors  should  provide  the  institute 
director  with  a  written  evaluation  of  the  scientific  director,  including  a  discus¬ 
sion  of  the  overall  direction  of  the  intramural  program  and  the  efficacy  of  the 
review  process.  The  institute  director  may  request  an  evaluation  of  the  scien¬ 
tific  director  by  the  chairperson  each  year. 

Boards  of  Scientific  Counselors 

3.  Nominations  for  the  board  of  scientific  counselors  should  be  provided  to  the 
director  of  the  institute  by  the  chairman  of  the  board  of  scientific  counselors, 
with  the  advice  of  the  scientific  director.  The  institute  director  and  the 
Deputy  Director  for  Intramural  Research,  NIH,  should  provide  a  final  review 
and  approval  of  the  list  of  nominees.  The  appointment  authority,  with  the 
exception  of  the  National  Cancer  Institute  (NCI),  should  remain  with  the 
Director,  NIH.  Ad  hoc  reviewers  should  be  selected  by  the  chairperson  of  the 
board  of  scientific  counselors,  with  the  advice  of  the  scientific  director. 
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4.  The  term  of  office  for  scientific  counselors  should  be  increased  to  5  years  to 
facilitate  continuity  of  review  and  to  allow  the  review  of  specific  programs 
more  than  once  by  the  same  counselor(s). 

5.  The  institutes  should,  to  the  extent  possible,  increase  the  number  of  indi¬ 
viduals  serving  on  their  boards  of  scientific  counselors. 

6.  Upon  completion  of  their  appointment,  scientific  counselors  should  be  sur¬ 
veyed  by  the  chairperson  of  the  board  of  scientific  counselors  on  issues  of 
program  quality  and  the  scientific  review  process.  This  survey  information 
should  be  provided  to  current  board  members. 

Review  Function  and  Process 

7.  Uniform  standards  for  scientific  review  procedures  should  be  developed  and 
codified. 

8.  Within  certain  specific  requirements,  each  institute  should  be  allowed  to 
develop  its  own  procedures  for  the  organization  and  structure  of  the  scientific 
review  meetings.  The  Deputy  Director  for  Intramural  Research,  NIH,  should 
evaluate  the  institutes'  procedures  to  ensure  that  the  uniform  requirements  of 
the  NIH  are  met  and  to  communicate  the  experiences  of  the  different  proc¬ 
esses  to  all  of  the  scientific  directors. 

9.  Scientific  counselors  should  be  provided  with  orientation  guidelines  that 
describe  the  review  procedures,  the  goals  of  the  process,  and  the  respon¬ 
sibilities  of  counselors.  These  guidelines  should  be  published  by  the  Deputy 
Director  of  Intramural  Research,  NIH. 

10.  The  boards  of  scientific  counselors  should  provide  evaluation  and  advice  on 
science,  specific  projects  (including  new  areas  of  development),  tenure  and 
promotion  decisions,  resource  allocation,  and  other  administrative  matters. 

11.  All  board  members  should  be  provided  with  a  summary  of  the  organization 
of  the  laboratory  to  be  reviewed;  the  various  sections;  all  personnel,  space 
usage,  operating  budget,  and  outside  contracts;  and  Cooperative  Research 
and  Development  Agreements.  Each  scientist  being  reviewed  should  supply 
details  of  ongoing  work,  a  curriculum  vitae,  reprints  of  the  three  to  five  most 
important  recent  publications,  and  an  analysis  of  the  amount  of  support  staff 
and  space  that  scientist  uses. 

12.  Before  the  board  adjourns,  an  oral  summary  of  the  review  should  be 
presented  to  the  scientific  director;  the  institute  director;  and  the  Deputy 
Director  for  Intramural  Research,  NIH;  or  their  designees. 

13.  Within  a  month  of  adjournment,  the  board  should  provide  a  written  report  of 
its  review  to  the  scientific  director.  Those  portions  of  the  report  that  pertain 
to  particular  laboratory/branch  chiefs  should  be  provided  to  them  by  the 
scientific  director.  The  scientific  director  and  the  laboratory/branch  chief 
should  discuss  the  evaluations  and  recommendations  in  the  report.  The 
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scientific  director  may  also  choose  to  discuss  appropriate  parts  of  the  report 
with  section  chiefs  and  individual  scientists  who  have  been  reviewed. 

14.  Within  6  months  of  receiving  the  board's  report,  but  prior  to  the  next  board  of 
scientific  counselors'  meeting,  the  scientific  director  should  respond  to  the 
board  indicating  areas  of  agreement  and  disagreement  and  any  possible 
changes  that  may  be  undertaken.  A  copy  of  the  report  and  the  response  of 
the  scientific  director  should  be  sent  to  the  director  of  the  institute;  the  Deputy 
Director  for  Intramural  Research,  NIH;  the  Director,  NIH;  and  the  institute's 
national  advisory  council. 

15.  No  later  than  1  year  after  the  review,  a  written  response  should  be  provided 
to  the  board  of  scientific  counselors  by  the  scientific  director,  summarizing 
the  results  of  any  actions  taken. 

16.  The  institute's  national  advisory  council  should  be  provided  an  overview  of 
the  institute's  intramural  research  program  at  least  once  a  year.  The  overview 
should  include,  at  a  minimum,  a  discussion  of  the  board  of  scientific 
counselors'  report  and  the  response  of  the  scientific  director. 
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Background 

During  the  development  of  the  FY 1989  President's  Budget,  the  OMB  requested 
DHHS  to  evaluate  strategies  to  strengthen  the  scientific  excellence  of  the  NIH 
intramural  laboratories,  through  an  independent  study  by  NAS/IOM.  In  Decem¬ 
ber  1988,  NAS/IOM  issued  its  report,  A  Healthy  NIH  Intramural  Program:  Struc¬ 
tural  Change  or  Administrative  Remedies?  (Excerpts  from  the  report  are  included  in 
appendix  A.) 

The  NAS/IOM  committee  found  that  "the  NIH  intramural  research  program  is  a 
distinctive  and  valuable  component"  of  the  Nation's  biomedical  research 
enterprise,  and  that  it  "continues  to  make  invaluable  contributions  to  our 
knowledge  and  understanding  of  basic  biological  processes  and  their  dysfunc¬ 
tion  in  disease."  The  committee  recommended  administrative  changes  in  the 
existing  intramural  research  program  structure  that  address  problems  associated 
with  increased  flexibility  in  personnel  administration,  maintaining  administra¬ 
tive  efficiency,  and  improving  the  review  of  the  intramural  research  program. 

Among  several  recommendations  that  were  intended  to  maintain  the  high  scien¬ 
tific  standards  of  NIH,  the  NAS/IOM  committee  suggested  two  that  it  felt  "essen¬ 
tial  to  credible  quality  assurance"  for  the  intramural  scientific  review  process: 

"A  panel  chaired  by  a  member  of  the  NIH  Director's  Advisory  Commit- 
^  tee  should  be  established  to  monitor  the  intramural  research  program 
review";  and, 

"Each  of  the  scientific  directors  and  their  intramural  programs  should  be 
reviewed  as  a  whole  every  four  years  by  an  external  group.  The  review 
u  report  should  be  submitted  to  the  director  of  the  relevant  institute,  the 
NIH  Deputy  Director  for  Intramural  Research,  the  Director  of  NIH, 
and  the  Director's  Advisory  Committee." 

Introduction 

In  response  to  these  two  recommendations,  the  Director  of  NIH  established  a 
Consultant  Panel  to  the  DAC  to  determine  whether  the  NAS/IOM 
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recommendations  might  strengthen  the  NIH  intramural  research  program  and,  if 
they  might,  how  they  could  be  implemented  most  effectively.  The  Director  also 
requested  the  Consultant  Panel  to  "examine  all  aspects  of  the  current  NIH  intra¬ 
mural  scientific  review  procedures  and  to  make  recommendations  for  strengthen¬ 
ing  the  intramural  review  process." 

A  meeting  of  the  Panel  was  held  at  NIH  on  September  5, 1989.  (See  appendixes 
B  and  C  for  the  agenda  and  a  list  of  presenters.)  The  Panel  addressed  the  two 
NAS/IOM  recommendations,  as  well  as  additional  issues  concerning  the 
intramural  scientific  review  procedures.  It  examined  the  procedures  for  the  selec¬ 
tion  of  reviewers  to  serve  on  boards  of  scientific  counselors,  the  mechanisms  that 
are  in  place  to  communicate  the  results  of  the  scientific  reviews  made  by  the 
boards,  and  the  implementation  of  their  recommendations.  The  Panel  also  was 
concerned  about  negative  perceptions  of  the  review  procedures,  particularly 
those  that  called  into  question  the  accountability  and  independence  of  the  intra¬ 
mural  review  process.  Thus  the  Panel  set  out  to  examine  fully  the  scientific 
review  procedures  of  the  NIH  intramural  program  and  to  develop  a  report  with 
recommendations  for  strengthening  the  review  process. 
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Information  Considered  by  the 
Consultant  Panel 

In  advance  of  the  meeting,  the  Panel  was  provided  with  briefing  materials  includ¬ 
ing  the  NAS/IOM  report;  Chapter  3005  of  the  DHHS  manual,  which  establishes 
the  policy  and  procedures  for  outside  review  and  evaluation  of  intramural 
research  at  NIH;  a  working  paper  prepared  by  NIH  staff  that  suggested  changes 
to  the  current  procedures;  a  documented  survey  and  analysis  of  intramural 
evaluation  prepared  by  the  NIH  Office  of  Intramural  Research;  and  a  side-by- 
side  comparison  of  procedures  in  Chapter  3005  and  the  proposals  in  the  working 
paper.  (See  appendixes  D,  E,  F,  and  G.) 

During  its  meeting  the  Panel  heard  testimony  on  the  intent  of  the  NAS/IOM 
study  committee's  recommendations  from  Dr.  Robert  Petersdorf,  a  member  of 
the  original  study  committee.  The  Panel  also  heard  presentations  from  Dr. 
Michael  Millman  (director  of  the  NAS/IOM  study)  and  several  senior  NIH 
scientists,  including  institute  directors  and  the  scientific  directors  of  the  intra¬ 
mural  research  programs.  The  following  sections  summarize  the  information 
considered  by  the  Panel. 

77 te  NAS/IOM  Report 

The  NAS/IOM  study  committee  reported  the  following  principal  findings  on  the 
NIH  intramural  research  program: 

•  "The  intramural  program  has  made  and  continues  to  make  invaluable  con¬ 
tributions  to  our  knowledge  and  understanding  of  basic  biological  processes 
and  their  dysfunction  in  disease." 

•  "A  high  quality  intramural  program  is  a  distinctive  and  valuable  component 
of  the  nation's  overall  biomedical  research  effort." 

•  'The  quality  of  the  program,  however,  varies  by  scientific  sub-field.  To 
improve  the  overall  quality  and  maintain  the  excellence  and  credibility  of  the 
program,  attention  must  be  paid  to  a  continuing  process  of  quality  review  and 
how  it  can  be  used  to  improve  the  allocation  of  resources." 

•  "The  scientific  directors  of  the  institutes,  who  most  directly  manage  the  intra¬ 
mural  program,  are  essential  keys  to  their  success.  Therefore,  finding  ways  to 
ensure  the  selection  and  retention  of  distinguished  scientific  leaders  for  these 
posts  is  essential." 

The  study  committee  also  found  that  despite  the  stature  of  the  scientific  coun¬ 
selors  and  ad  hoc  consultants  that  serve  in  the  NIH  review  process,  the  "fact  that 
they  are  nominated  by  the  scientific  director  of  the  institute  whose  program  will 
be  under  review  compromises  the  external  credibility  of  their  report."  Also, 
despite  recent  improvements  made  by  NIH  through  feedback  mechanisms  to  the 
board  of  scientific  counselors  on  the  implementation  of  their  recommendations, 
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the  study  committee  felt  that  “the  scientific  director  is  at  no  time  accountable  to 
anyone  outside  the  institute." 

These  concerns  led  to  the  two  principal  NAS/IOM  recommendations  to 
strengthen  the  NIH  intramural  review  process:  first,  the  creation  of  a  monitoring 
panel  chaired  by  a  member  of  the  DAC;  and  second,  that  the  scientific  directors 
and  their  intramural  programs  should  be  reviewed  as  a  whole  every  4  years  by 
an  external  group. 

According  to  the  NAS/IOM  report,  the  proposed  monitoring  panel's  function 
would  be  to  oversee  the  integrity  of  the  review  process,  not  to  replicate  the  activ¬ 
ities  of  the  boards  of  scientific  counselors.  The  oversight  function  “should  focus 
on  areas  that  are  most  vulnerable  to  criticism,  namely  the  selection  of  the 
reviewers  and  the  appropriate  response  to  recommendations." 

The  NAS/IOM  study  committee  also  believed  that  the  proposed  quadrennial 
review  of  the  scientific  directors  and  their  intramural  programs  was  necessary 
“because  of  the  importance  of  ensuring  the  vitality  of  the  intramural  program  . . . 
through  a  process  that  would  ensure  vigorous  leadership." 

The  role  of  the  scientific  directors  was  seen  as  vital  to  the  success  of  the  intra¬ 
mural  programs,  as  reflected  in  the  following  recommendation  in  the  NAS/IOM 
report: 

“. . .  the  responsibility  of  the  scientific  director  requires  having  the 
scientific  vision  needed  to  allocate  intramural  resources  productively, 
as  well  as  function  as  a  highly  skilled  manager.  To  recruit  and  retain 
scientists  with  this  extraordinary  set  of  attributes,  the  committee  recom¬ 
mends  that  those  holding  the  position  of  scientific  director  receive  addi¬ 
tional  compensation." 

Testimony  of  Dr.  Petersdorf 

In  his  presentation  to  the  Panel,  Dr.  Petersdorf  (an  NAS/IOM  study  committee 
member)  indicated  that  a  key  issue  for  the  NAS/IOM  study  committee  had  been 
how  the  review  process  of  the  intramural  research  program  is  perceived  by  indi¬ 
viduals  outside  NIH.  Those  perceptions  include  the  possibility  of  less  rigorous 
review  of  intramural  than  extramural  research,  unevenness  of  intramural 
reviews  among  NIH  institutes,  inadequate  usage  of  reviews,  and  the  potential 
conflict  of  interest  in  selecting  scientific  counselors.  Dr.  Petersdorf  indicated  that 
the  NAS/IOM  study  committee  also  had  been  left  with  the  impression  that  the 
boards  of  scientific  counselors  do  not  have  a  modus  operandi  that  can  be 
described  easily,  and  that  there  is  considerable  variability  in  how  the  review 
process  is  pursued. 

The  NAS/IOM  committee  sought  the  institutionalization  of  a  mechanism  for 
external  peer  review  of  intramural  research  that  would  function  at  arm's  length 
from  the  officials  whose  programs  were  under  review  to  eliminate  the  possibility 
of  conflict  of  interest  (real  or  apparent)  in  the  behavior  of  members  of  the  external 
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review  bodies.  The  goal  embodied  in  the  NAS/IOM  committee  recommendations 
was  that  the  direct  responsibility  for,  and  control  of,  the  process  of  external 
evaluation  of  intramural  research  be  transferred  from  the  scientific  directors  to 
the  Director  of  NUT. 

The  NAS/IOM  study  committee  also  believed  that  any  changes  in  the  current 
intramural  program  review  procedures  must  be  made  carefully.  The  organiza¬ 
tion,  structure,  and  location  of  the  intramural  research  program,  for  example,  are 
unique  and  require  a  system  of  peer  review  that  is  adapted  to  the  program's  spe¬ 
cial  characteristics  and  that  differs  from  the  process  in  place  for  project-based 
extramural  research. 

DHHS  Manual  Chapter  3005 

Intramural  research  at  NIH  has  been  reviewed  by  committees  of  scientists  from 
outside  NIH  since  1956.  In  that  year  the  first  committees,  called  boards  of  scien¬ 
tific  counselors,  were  appointed  to  review  intramural  research  in  the  individual 
institutes.  Board  procedures  have  evolved  over  the  years.  On  March  4, 1985,  the 
original  version  of  DHHS  Manual  Chapter  3005  (" Review  and  Evaluation  of  Intra¬ 
mural  Research  by  Boards  of  Scientific  Counselors  “)  was  issued  to  establish  policy 
and  procedures  for  the  evaluation  of  intramural  research  by  the  boards.  Subse¬ 
quently,  Public  Law  99-158,  the  Health  Research  Extension  Act  of  1985,  was 
enacted  (on  November  20, 1985).  It  included  a  provision  making  mandatory 
what  was  then  standard  policy  on  review  of  intramural  research. 

The  boards  of  scientific  counselors  are  public  advisory  committees  established 
under  provisions  of  the  Federal  Advisory  Committee  Act  (Public  Law  92-463). 
The  boards  review,  discuss,  and  evaluate  research  projects  and  programs  con¬ 
ducted  by  NIH  and  by  ADAMHA  intramural  programs  located  on  the  NIH 
campus.  The  reviews  include  evaluation  of  the  work  of  individual  scientists  as 
well  as  the  quality  of  the  research  itself. 

It  is  the  policy  of  NIH  that  the  intramural  programs  of  the  agency  be  reviewed 
regularly  by  highly  qualified  outside  scientists.  The  reviews  include  the  quality 
of  the  research  and  the  accomplishments  of  individual  scientists.  This  policy  is 
reinforced  by  section  492(b)  of  the  Public  Health  Service  (PHS)  Act: 

The  Director  of  NTH  shall  establish  procedures  for  periodic  technical 
and  scientific  peer  review  of  research  at  the  National  Institutes  of 
Health.  Such  procedures  shall  require  that  — 

1.  the  reviewing  entity  be  provided  a  written  description  of  the 
research  to  be  reviewed,  and 

2.  the  reviewing  entity  provide  the  advisory  council  of  the  national 
research  institute  involved  with  such  description  and  the  results  of 
the  review  by  the  entity. 
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These  reviews,  and  any  resulting  recommendations,  should  be  included  in  a  writ¬ 
ten  report  submitted  to  the  scientific  director,  who  will  transmit  it  to  the  institute 
director;  the  NIH  Deputy  Director  for  Intramural  Research;  and  the  Director, 

NIH.  In  addition,  as  required  by  the  PHS  Act,  the  institute  will  provide  written 
descriptions  of  the  research  reviewed  by  the  board  of  scientific  counselors,  and 
results  of  that  review  to  its  national  advisory  council.  Such  communications 
should  be  made  to  the  council  at  least  annually.  Also,  as  stated  in  the  PHS  Act, 
the  advisory  council  "may  on  the  basis  of  the  materials  provided  under  Section 
492(b)(2)  respecting  research  conducted  at  the  institute,  make  recommendations 
to  the  Director  of  the  institute  respecting  such  research." 

The  Chapter  3005  procedures  require  that  the  "primary  consideration  (for)  scien¬ 
tific  qualification  for  board  membership  shall  be  international  recognition  as  an 
authority  in  the  fields  under  review."  The  section  also  requires  each  board  to 
exhibit  reasonable  balance  in  membership  with  respect  to  gender,  race,  and 
geographical  distribution  of  its  member  institutions.  Board  members  are 
nominated  by  the  institute  scientific  director  and,  with  the  exception  of  NCI, 
approved  by  the  institute  director  and  the  Deputy  Director  for  Intramural 
Research,  NIH,  and  appointed  by  the  Director,  NIH.  Under  the  PHS  Act,  the 
Director,  NCI,  has  authority,  in  consultation  with  the  National  Cancer  Advisory 
Board,  to  appoint  members  of  NCI  Boards  of  Scientific  Counselors. 

Chapter  3005  procedures  also  require  boards  to  "meet  often  enough  (ordinarily 
two  or  three  times  each  year)  to  assure  that  the  work  of  each  intramural  scientist 
on  a  permanent  appointment  in  each  laboratory  or  branch  is  reviewed  at  least 
once  every  four  years."  Each  scientific  director  must  submit  to  the  Deputy  Direc¬ 
tor  for  Intramural  Research,  NIH,  a  schedule  of  the  proposed  dates  of  review  of 
each  laboratory  or  branch  in  the  institute  or  division  covering  the  next  4-year 
review  cycle.  This  schedule  must  be  updated  annually  and  submitted  by 
September  1  each  year. 

The  following  information  must  be  provided  to  each  board  reviewer  in  advance 
of  the  meeting: 

•  For  each  laboratory  or  branch  being  reviewed — a  staff  list  of  all  employees, 
including  name,  occupational  specialty,  type  of  appointment  and  grade;  and  a 
list  of  all  contracts  that  support  the  work  of  the  laboratory  or  branch. 

•  For  each  scientist  whose  work  is  to  be  reviewed — a  current  curriculum  vitae, 
bibliography,  and  recent  relevant  reprints;  a  succinct  summary  of  current 
research  and  statement  of  future  plans;  and  the  amount  of  technical  and 
postdoctoral  support  required  by  each  scientist. 

A  copy  of  the  last  review  of  the  laboratory  or  branch  should  be  available  to  the 
board.  The  conference  report  for  this  section  of  the  PHS  Act  notes  that  "the  writ¬ 
ten  description  should  present  the  overall  past  accomplishments  of  the  labor¬ 
atory  and  its  investigators  since  the  last  review  was  conducted.  The  written 
description  might  also  present  the  general  aims,  the  objectives,  and  the  projected 
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directions  of  the  research  projects  to  be  conducted;  appropriate  background  infor¬ 
mation  on  possible  future  research  projects;  a  brief  summary  of  the  general 
research  plan;  and  an  indication  of  the  importance  of  the  research  projects. 

Every  independent  intramural  scientist  on  a  permanent  appointment  must  be 
reviewed  and  evaluated.  In  general,  the  advice  of  the  board  of  scientific  coun¬ 
selors  also  should  be  obtained  concerning  scientists  who  are  being  considered  for 
conversion  to  tenure,  and  the  scientific  director  may  ask  the  board  to  review  the 
work  of  other  staff  scientists.  Over  time,  the  active  program  of  each  laboratory  or 
other  intramural  research  unit  is  to  be  reviewed. 

The  organization  and  structure  of  meetings  of  the  board  of  scientific  counselors  is 
left  to  the  discretion  of  the  scientific  director,  but  the  reviewers  should  be  given 
an  opportunity  to  visit  the  facilities  where  the  research  is  carried  out. 

Chapter  3005  sets  forth  the  provisions  to  be  followed  in  reporting  the  results  of 
the  reviews.  The  report  of  the  board  is  to  be  a  narrative  critique  from  the  chair¬ 
person  of  the  board,  following  the  outline  preferred  by  the  scientific  director. 
Evaluations  of  individual  investigators  must  address,  at  a  minimum,  the  follow¬ 
ing  questions:  are  good  research  questions  being  asked,  are  appropriate 
approaches  being  used  to  obtain  answers,  and  are  the  resources  available  to  this 
scientist  appropriate  to  the  accomplishments  to  date?  These  evaluations  should 
be  written  by  members  of  the  board  of  scientific  counselors  or  consultants  and 
should  reflect  the  collective  views  of  the  total  board  (or  at  least  a  majority). 

Within  4  months  of  the  review,  the  report  is  to  be  sent  to  the  scientific  director, 
who  is  responsible  for  transmitting  it  to  the  institute  director;  the  NIH  Deputy 
Director  for  Intramural  Research;  and  the  Director,  NIH.  The  Office  of  Intra¬ 
mural  Affairs  distributes  copies  to  the  NIH  Board  of  Scientific  Directors  and 
schedules  its  discussion  by  all  members  of  that  group. 

At  least  annually,  each  institute  transmits  descriptions  of  the  research  that  has 
been  reviewed  by  its  board(s)  of  scientific  counselors  during  the  preceding  year 
to  its  national  advisory  council  together  with  the  results  of  the  reviews.  The 
results  of  the  reviews  and  board  recommendations  may  be  reported  to  the  coun¬ 
cil  orally  or  in  writing  by  the  scientific  director  or  the  chairperson  of  the  board. 

In  written  reports,  the  evaluation  of  individual  scientists  may,  for  privacy 
reasons,  be  omitted. 

It  is  the  responsibility  of  the  scientific  director  to  report  to  the  board  of  scientific 
counselors  the  action  taken  in  response  to  recommendations  made  by  the  board. 
This  report  shall  describe  actions  taken  or  explain  why  it  was  found  impossible 
or  inadvisable  to  follow  the  recommendations. 
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Overview  ofNIH  Intramural  Scientific  Review  Procedures 

Dr.  Joseph  E.  Rail,  Deputy  Director  for  Intramural  Research,  NIH,  provided  the 
Panel  with  comments  about  the  NIH  intramural  scientific  review  procedures, 
including  the  following  key  points: 

•  The  relationship  of  a  board  of  scientific  counselors  to  the  intramural  research 
program  it  reviews  is  much  different  today  than  it  was  in  the  1970's. 

•  Institutes  differ  somewhat  in  how  they  use  their  board  of  scientific  counselors, 
in  part  because  of  organizational  differences.  Such  an  arrangement  is  valu¬ 
able  because  it  enables  each  institute  to  learn  from  others. 

•  Only  distinguished  and  highly  recognized  scientists  are  considered  for  board 
appointments,  and  those  appointments  must  be  approved  by  the  Deputy 
Director  for  Intramural  Research,  NIH,  (or  in  the  case  of  NCI,  by  the  Director, 
NCI).  Occasionally  the  Deputy  Director  for  Intramural  Research,  NIH,  sug¬ 
gests  changes  in  the  appointment  of  members. 

•  Scientific  counselors  provide  scientific  advice  as  well  as  recommendations  on 
promotion  and  tenure,  the  best  use  of  available  resources,  whether  a  partic¬ 
ular  laboratory  should  receive  support,  and  certain  administrative  issues  such 
as  the  length  of  a  staff  fellowship. 

•  Boards  of  scientific  counselors  are  restricted  from  certain  activities.  For 
example,  the  boards  can  neither  increase  salaries  of  outstanding  scientists  nor 
directly  hire  outstanding  scientists  to  develop  new  areas  of  the  intramural 
program. 

•  Each  scientific  director  pays  careful  attention  to  suggestions  from  the  board, 
which  helps  strengthen  his  or  her  position  in  directing  the  scientific  programs 
of  the  institute. 

•  Although  the  salary  of  a  scientist  in  the  intramural  program  cannot  be  sum¬ 
marily  withdrawn,  laboratory  funding  and  support  can.  Scientists  in  the 
intramural  program,  like  their  counterparts  in  the  extramural  program,  must 
compete  for  resources  and  are  constantly  required  to  do  so. 

The  intramural  program  is,  in  certain  ways,  unique.  For  a  scientist,  becoming  a 
tenured  member  of  the  program  is  a  process  that  takes  many  years,  during  which 
time  programs  and  resources  are  developed.  During  this  time,  candidates  have 
the  chance  to  demonstrate  outstanding  research  competence.  In  this  environ¬ 
ment,  both  tenure  candidates  and  tenured  scientists  have  the  opportunity  to 
address  difficult,  long-term  research  goals  that  may  not  produce  immediate 
results.  Progress  toward  these  goals,  as  well  as  the  scientific  achievements  of 
each  individual  scientist,  are  continually  reviewed.  The  institute,  however,  seeks 
to  preserve  the  long-term  goals  of  each  designated  program  that  may  involve 
many  scientists,  as  long  as  those  goals  remain  appropriate.  The  research  results 
that  have  emanated  from  this  approach  are  evident.  Dr.  Rail  suggested  that  any 
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attempt  to  improve  the  review  process  should  be  accepted  only  if  it  would 
clearly  improve  the  quality  of  intramural  research. 

Unique  Institute  Characteristics 

National  Institute  of  Allergy  and  Infectious  Diseases  (NIAID) 

Dr.  John  Gallin,  Director  of  the  Intramural  Research  Program  in  NIAID, 
provided  the  following  comments  on  how  intramural  research  is  reviewed  in 
that  institute: 

•  The  board,  which  meets  twice  a  year,  consists  of  eight  scientific  counselors 
and  eight  ad  hoc  reviewers.  The  members  of  the  board  are  nominated  by  the 
Scientific  Director,  after  staff  consultation  with  the  Director,  NIAID,  to  the 
Deputy  Director  for  Intramural  Research,  NIH,  who  approves  the 
appointments. 

•  A  chairperson,  chosen  by  the  scientific  director  from  among  those  scientific 
counselors  serving  for  a  third  or  fourth  year,  selects  the  ad  hoc  reviewers  for 
the  upcoming  meeting  from  a  list  drawn  up  by  the  scientific  director,  after 
staff  consultations.  The  chairperson  of  the  board  of  scientific  counselors  may 
add  to  or  delete  from  the  list  of  names. 

•  Participants  receive,  in  advance,  a  briefing  book  from  the  scientific  director 
describing  the  review  process  and  a  book  for  each  laboratory  under  review. 
The  latter  book  reviews  past  laboratory  accomplishments;  future  directions; 
and  administrative  details  on  the  space,  support  personnel,  contracts,  and — 
starting  in  December  1989 — direct  operating  costs.  Board  members  are 
provided  with  primary  review  assignments  before  arriving  at  NIH. 

•  At  each  year's  initial  session  of  the  board  of  scientific  counselors,  the  scientific 
director  goes  over  the  entire  review  process.  Each  laboratory  is  reviewed 
every  4  years.  Following  the  opening  session  each  laboratory  chief  makes  an 
oral  presentation,  followed  by  presentations  from  selected  senior  scientists 
and  individual  candidates  for  tenure. 

•  Each  board  member  meets  privately  with  individual  scientists,  and  each 
scientist  has  at  least  two  interviews.  However,  every  candidate  for  tenure, 
laboratory  chief,  and  section  head  has  three  interviews.  These  interviews  last 
30  minutes  to  an  hour  and  are  held  in  the  laboratory,  where  notebooks  and 
other  materials  are  available  at  the  interviewer's  request.  Each  board  member 
then  writes  a  250-  to  500-word  evaluation  of  each  scientist  interviewed. 

•  The  board  also  approves  concepts  for  new  contracts  and  for  contracts  up  for 
renewal.  The  scientific  director  does  not  attend  the  board's  closed  executive 
session,  so  that  the  board  may  discuss  the  scientists  interviewed  and  draw  up 
final  recommendations.  In  addition,  selected  board  members  write  overviews 
of  each  section  and  laboratory  reviewed  at  that  session. 
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•  In  closed  executive  session  the  board  discusses  the  laboratory  summary  over¬ 
views  and  provides  each  laboratory  chief  with  a  private  evaluation,  allows  the 
individual  to  comment,  and  revises  its  evaluation  if  necessary. 

•  At  the  end  of  its  meeting,  the  board  also  meets  and  discusses  the  reviews  and 
its  recommendations  with  the  NIAID  Director;  the  scientific  director;  and  the 
Deputy  Director  for  Intramural  Research,  NIH,  or  his  designee. 

•  Before  adjourning,  the  board  submits  all  written  reports  to  the  scientific  direc¬ 
tor  and  later  sends  back  comments  and  evaluations  on  the  review  process 
itself. 

•  Within  a  month  after  the  meeting  ends,  the  final  report  is  assembled.  The 
scientific  director  then  provides  each  scientist  interviewed  with  the 
anonymous  reports  of  the  board  members,  discusses  the  review  process,  and 
gives  an  impression  and  evaluation  of  the  scientist.  The  scientific  director 
provides  each  section  chief  and  laboratory  chief  with  an  evaluation  of  their 
scientific  staff  and  the  overview  evaluation  of  their  section/laboratory. 

•  Within  6  months  of  the  last  board  meeting,  the  scientific  director  provides  the 
board  with  an  oral  report,  and  within  a  year  a  written  report,  that  documents 
the  institute's  followup  activities  in  response  to  the  board's  recommendations. 
A  copy  of  this  summary  board  report  is  transmitted  to  the  Director  of  NIAID, 
the  NIH  Deputy  Director  for  Intramural  Research,  and  the  NIH  Board  of 
Scientific  Directors  within  6  months  of  the  review.  Once  a  year  the  NIAID 
National  Advisory  Council  is  briefed  on  the  board's  activities  and  the  state  of 
the  institute's  intramural  program. 

Previous  discussions  among  the  NIAID  Director,  Scientific  Director,  and  Chair¬ 
man  of  the  NIAID  Board  of  Scientific  Counselors  led  to  the  conclusion  that 
although  a  formal  review  of  the  scientific  director  was  unlikely  to  work,  the  scien¬ 
tific  director  could  provide  either  an  oral  or  a  written  presentation  to  the  board 
about  the  institute's  work  and  how  resources  are  allocated.  An  added  recommen¬ 
dation  was  that  once  every  4  years  the  chairperson  of  the  board  of  scientific  coun¬ 
selors  should  prepare  a  consensus  report,  based  on  the  board's  views,  of  the  state 
of  the  institute's  intramural  program  and  the  review  process  undertaken  by  the 
board. 

National  Institute  of  Child  Health  and  Human  Development  (NICHD) 

Dr.  Arthur  Levine,  Scientific  Director,  emphasized  the  following  key  points  for 
review  of  intramural  research  at  NICHD: 

•  The  diversity  of  scientific  research  in  NICHD  requires  a  nine-member  board 
of  scientific  counselors. 

•  Before  each  board  meeting,  board  members  receive  a  briefing  book  with  infor¬ 
mation  on  overall  resource  allocation  and  current  scientific  accomplishments. 
The  book  contains  reports  of  all  site  visits  held  since  the  previous  board  meeting. 
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•  At  twice-yearly  board  meetings,  the  scientific  counselors  review  the  institute's 
entire  intramural  program  rather  than  specific  laboratories. 

•  The  board  also  reviews  actual  site  visit  reports  prepared  since  the  last  meet¬ 
ing,  followup  reports  to  previous  visits,  and  oral  presentations  by  tenure  can¬ 
didates.  In  addition,  generic  issues  such  as  policies  for  the  review  of 
intramural  research,  scientific  misconduct,  and  animal  care  are  discussed. 

•  Three  laboratory  site  visits  are  conducted  each  year,  each  chaired  by  a  dif¬ 
ferent  board  member,  resulting  in  a  review  of  each  of  the  institute's 
laboratories  and  branches  once  every  4  years.  Often  two  or  three  other  board 
members  will  participate  in  a  site  visit,  together  with  up  to  10  or  12  ad  hoc 
visitors.  The  chairperson  for  the  visit  selects  ad  hoc  reviewers  from  a  list 
prepared  by  the  scientific  director  and  may  add  and  delete  names. 

•  Site  reviewers  receive  a  briefing  book  prepared  by  the  laboratory  and  an  orien¬ 
tation  by  the  scientific  director  before  the  site  visit  takes  place.  This  book  con¬ 
tains  detailed  descriptions  of  the  results  of  each  tenured  scientist's  research, 
current  work  (goals,  methods,  etc.),  and  proposed  future  research.  A  detailed 
accounting  of  resources  is  also  provided,  as  well  as  curriculum  vitae, 
preprints,  protocols,  etc. 

•  The  site  visit,  which  lasts  2  days  for  each  laboratory  or  branch,  consists  of 
1-hour  presentations  by  each  tenured  or  tenure-candidate  scientist. 

Reviewers  also  visit  the  laboratories,  where  they  meet  and  talk  with  the  staff 
and  where  they  can  view  special  techniques,  equipment,  and  laboratory 
notebooks.  Site  reviewers  meet  with  the  laboratory  chief  and  the  scientific 
director  to  exchange  views  and  impressions,  and  they  draft  the  report  before 
adjourning. 

•  The  final  report  is  sent  to  the  scientific  director,  who  transmits  it  to,  and  dis¬ 
cusses  it  with,  the  laboratory  chief;  the  laboratory  chief  in  turn  transmits 
relevant  sections  of  the  report  to  each  investigator  in  the  laboratory.  A  copy 
of  the  report  is  forwarded  to  the  Director  of  NICHD;  the  Deputy  Director  for 
Intramural  Research,  NIH;  and  the  other  NIH  scientific  directors. 

•  At  the  next  semiannual  meeting  of  the  board  of  scientific  counselors,  the 
report  is  distributed  for  discussion  together  with  the  written  response  of  the 
laboratory  chief. 

•  The  scientific  director  provides  the  board  with  followup  information  6  months 
later,  i.e.,  1  year  after  the  visit.  In  addition,  once  a  year  the  chairperson  of  the 
board  of  scientific  counselors  makes  a  report  to  the  NICHD  National  Ad¬ 
visory  Council  on  the  review  process  and  the  main  conclusions  of  that  year's 
reviews. 
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National  Cancer  Institute  (NCI) 

Dr.  Alan  Rabson,  Director  of  the  Division  of  Cancer  Biology  and  Diagnosis, 

presented  the  following  highlights  of  intramural  research  review  at  NCI: 

©  Differences  in  the  scientific  review  of  intramural  research  programs  at  NCI 
can  be  attributed  to  the  different  organizational  structure  of  the  Institute. 
There  is  no  separate  division  of  intramural  research  at  NCI;  instead,  the  intra¬ 
mural  research  programs  are  located  within  three  of  the  operating  divisions — 
the  Division  of  Cancer  Biology  and  Diagnosis,  the  Division  of  Cancer 
Etiology,  and  the  Division  of  Cancer  Treatment.  Each  of  these  divisions  also 
has  an  active  extramural  research  program.  The  director  of  each  division 
serves  as  the  scientific  director  of  both  the  extramural  and  the  intramural 
research  programs.  Each  division  has  its  own  board  of  scientific  counselors 
that  advises  the  division  director  on  both  intramural  and  extramural  research 
programs.  In  the  extramural  program,  the  board  is  responsible  for  concept 
approval  of  all  program  initiatives  such  as  Requests  for  Applications  and 
Program  Announcements  for  grants  and  Requests  for  Proposals  for  contracts. 

©  The  NCI  divisional  boards  of  scientific  counselors  are  larger  than  those  of 
other  institutes,  ranging  from  15  to  20  members.  The  members  are  appointed 
by  the  Director,  NCI.  Because  their  responsibilities  also  include  the  extramural 
program,  it  would  be  inappropriate  for  the  board  members  to  be  approved  by 
the  Deputy  Director  for  Intramural  Research,  NIH;  however,  he  is  provided 
with  an  informational  copy  of  the  list  of  proposed  board  members. 

•  Because  the  NCI  boards  are  larger  than  those  of  the  other  institutes  and  must 
have  members  with  a  broad  range  of  expertise  to  review  extramural  as  well  as 
intramural  programs,  it  is  not  always  appropriate  for  all  of  the  members  to 
attend  each  site  visit.  The  board  chairperson,  with  the  advice  of  the  Division 
Director,  appoints  a  board  member  with  appropriate  expertise  to  be  chairper¬ 
son  of  the  site  visit  team.  Other  board  members  with  appropriate  scientific 
expertise  also  participate  in  the  site  visit.  Additional  ad  hoc  members  with 
special  expertise  in  the  research  programs  of  the  laboratory  being  reviewed 
are  selected  by  the  site  visit  chairperson,  with  advice  from  the  Division  Direc¬ 
tor  and  the  chairperson  of  the  full  board.  The  site  visit  chairperson  prepares 
the  report  and  sends  it  to  the  Division  Director.  The  report  is  then  presented 
and  discussed  by  the  full  board  in  closed  executive  session  at  the  first 
scheduled  meeting  following  the  site  visit.  One  year  after  the  site  visit,  the 
Division  Director  reports  back  to  the  board  on  the  actions  taken  in  response  to 
the  site  visit  report. 

Working  Paper  on  Proposed  NIH  Intramural 
Scientific  Review  Procedures 

Dr.  Edward  Korn,  Director  of  the  Division  of  Intramural  Research  of  the  National 

Heart,  Lung,  and  Blood  Institute  (NHLBI),  presented  highlights  of  a  working 
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paper  prepared  by  representatives  of  the  NIH  intramural  research  program.  This 
working  paper  provided  the  panel  with  a  synopsis  of  current  procedures  and 
suggestions  for  improving  the  review  process.  It  includes  the  following  key 
points  not  already  stated  explicitly  in  Chapter  3005: 

Composition  of  Boards 

•  The  board  of  counselors  should  consist  of  six  to  eight  members,  each  serving  a 
4-year  term. 

•  The  chairperson  of  the  board  of  scientific  counselors  will  be  selected  by  the 
scientific  director,  with  the  advice  of  the  institute  director,  usually  from  one  of 
the  members  serving  for  a  fourth  year. 

•  Nomination  of  board  members  will  be  conducted  with  the  advice  of  the  chair¬ 
person  of  the  board  of  scientific  counselors. 

•  The  board  will  be  supplemented  for  each  review  with  ad  hoc  reviewers 
selected  by  the  chairperson  of  the  board  with  the  advice  of  the  scientific 
director. 

Frequency  of  Review  Meetings 

•  Scientific  reviews  by  the  board  of  scientific  counselors  supplemented  by  ad 
hoc  reviewers  must  be  held  at  least  semiannually. 

Information  Supplied  to  Boards  of  Scientific  Counselors  Before  Meeting 

•  For  each  scientist  to  be  reviewed,  the  briefing  materials  must  also  include  a 
written  summary  of  the  presentation  (emphasizing  accomplishments  since  the 
last  review)  and  space  assignments. 

•  For  each  laboratory/branch  being  reviewed,  the  material  must  also  include  a 
summary  of  administrative  organization,  including  space  allocations,  budget 
information,  and  Cooperative  Research  and  Development  Agreements. 

Responsibilities  of  the  Board  of  Scientific  Counselors 

•  The  board  of  scientific  counselors  is  responsible  for  recommending  both 
programmatic  changes  and  changes  in  level  of  support  for  individual  scien¬ 
tists,  including  conversions  to  tenure  status  and  promotion. 

•  All  conversions  to  tenure  and  promotions  of  the  scientific  staff  within  any  one 
institute  must  be  approved  by  the  institute  director  and  the  NIH  Board  of 
Scientific  Directors. 

•  The  board  of  scientific  counselors  must  also  review  and  assess  the  perform¬ 
ance  of  the  scientific  director,  primarily  based  on  reviews  of  the  laboratories 
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and  branches.  In  addition,  at  periodic  intervals,  the  board  will  specifically 
review  the  scientific  director  and  make  a  formal  report  to  the  institute  director. 

Review  Format 

•  For  the  review  process,  the  laboratory/branch  chief  will  present  an  overview 
of  the  research  activities.  The  review  of  individual  scientists  may  be  oral 
presentations  to  the  assembled  board  followed  by  discussion,  or  a  mixture  of 
oral  presentations  and  separate  interviews  with  one  or  more  members  of  the 
board  and  ad  hoc  reviewers. 

•  The  board  will  hold  executive  sessions  to  discuss  the  merits  of  the  individual 
presentations  as  well  as  broader  aspects  of  the  intramural  program. 

•  The  board  may  wish  to  meet  with  the  laboratory/branch  chiefs  to  clarify  its 
understanding  of  the  scientific  presentations  and  to  provide  the  laboratory/ 
branch  chiefs  with  an  opportunity  to  respond  to  its  questions  before  the  final 
report  is  prepared. 

Reporting  of  Results  of  Reviews 

•  Before  the  board  adjourns,  an  oral  summary  will  be  presented  to  the  scientific 
director.  The  institute  director  and  the  Deputy  Director  for  Intramural 
Research,  NIH,  or  their  designees  will  be  invited  to  these  sessions.  The  board 
will  provide  a  written  report  to  the  scientific  director  within  1  month  of  the 
review. 

Followup 

•  The  portion  of  the  board's  report  that  pertains  to  particular  laboratory/branch 
chiefs  will  be  provided  to  them  by  the  scientific  director.  The  scientific  direc¬ 
tor  and  laboratory/branch  chief  will  discuss  the  evaluations  and  recommenda¬ 
tions  in  the  report.  The  scientific  director  may  also  wish  to  discuss  the  report 
privately  with  one  or  more  investigators  under  review. 

•  Within  6  months  of  receiving  the  report,  the  scientific  director  will  respond  to 
the  board,  indicating  areas  of  agreement  and  disagreement  and  any  possible 
changes  that  may  be  undertaken. 

•  The  scientific  director  will  transmit  copies  of  the  report  and  the  scientific 
director's  response  to  the  institute  director;  the  NIH  Deputy  Director  for  Intra¬ 
mural  Research;  and  the  Director,  NIH,  within  6  months  of  the  review.  The 
report  will  be  distributed  to  the  scientific  directors  of  all  the  institutes  and  be 
discussed  at  a  regularly  scheduled  meeting  of  the  NIH  Board  of  Scientific 
Directors. 

®  After  1  year,  the  scientific  director  will  respond  to  the  board  in  writing,  sum¬ 
marizing  the  results  of  any  actions  taken. 
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•  The  written  material  reviewed  by  the  board  the  previous  year  and  the  board's 
reports  will  be  presented  annually  to  the  institute's  national  advisory  council 
by  the  scientific  director. 

Review  of  the  Scientific  Director 

•  Once  every  4  years,  the  chairperson  of  the  board  of  scientific  counselors  will 
provide  a  written  report  to  the  institute  director,  evaluating  the  review 
process  and  the  overall  direction  and  major  accomplishments  of  the  intra¬ 
mural  program.  The  institute  director  may  request  an  evaluation  of  the  scien¬ 
tific  director  by  the  chairperson  each  year. 
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Deliberations  and  Findings  of  the 
Consultant  Panel 

The  Panel  addressed  the  two  specific  recommendations  of  the  NAS/IOM  study 
committee,  seeking  to  determine  if  the  recommendations  were  appropriate  for 
strengthening  the  intramural  review  procedures  and,  if  so,  how  best  to  imple¬ 
ment  them.  In  addition,  the  Panel  examined  all  aspects  of  the  current  scientific 
review  procedures  and  considered  ways  for  strengthening  the  process.  After  an 
extensive  discussion  of  these  points,  the  Panel  outlined  its  full  recommendations 
for  changing  the  scientific  review  procedures  of  the  NIH  intramural  research 
program. 

Monitoring  the  Intramural  Research 
Program  Review 

The  Consultant  Panel  held  a  far-reaching  discussion  on  the  NAS/IOM  recom¬ 
mendation  that  would  establish  a  panel,  chaired  by  a  DAC  member,  to  monitor 
the  intramural  research  program  review.  The  Consultant  Panel  recognized  that 
the  function  of  the  proposed  group  would  be  to  ensure  the  integrity  of  the  process, 
not  to  replicate  the  activities  of  the  boards  of  scientific  counselors.  The  Panel  was 
also  aware  that  a  principal  concern  of  the  NAS/IOM  study  committee  was  the  selec¬ 
tion  of  reviewers  and  the  implementation  of  the  boards'  recommendations. 

The  discussion  of  this  issue  began  with  a  review  of  the  structure  and  function  of 
the  DAC,  which  was  described  as  consisting  of  the  committee  chair  and  16 
appointed  members.  Eleven  members  are  authorities  knowledgeable  in  the 
fields  of  research  pertinent  to  the  NIH  mission,  and  five  are  representatives  of  the 
general  public.  The  DAC  provides  advice  on  policy  matters  pertinent  to  the  NIH 
mission  and  makes  recommendations  concerning  program  development,  NIH 
administrative  regulation  and  policy,  and  other  specific  or  general  aspects  of 
NIH  policy. 

The  ensuing  discussion  elaborated  on  factors  in  favor  of  accepting  the  recommen¬ 
dation.  The  point  was  made  that  a  standing  panel  of  the  DAC  could  meet 
periodically  to  assess  the  quality  issues  in  the  institutes  by  reviewing  the  appoint¬ 
ments  and  reports  of  various  boards  in  an  orderly  way.  Individual  members  of 
the  monitoring  panel  could  be  assigned  institutes  or  divisions  to  monitor  and 
serve  as  a  channel  to  the  DAC  to  keep  it  abreast  of  accomplishments  and 
problems.  The  panel,  as  envisioned,  would  report  problems  discovered  in  the 
review  process  to  the  Director,  NIH,  and  thereby  bring  about  a  uniformity  in 
review — a  condition  that  was  of  concern  to  the  NAS/IOM  study  committee. 

In  counterpoint,  questions  were  raised  about  the  operational  nature  of  the 
proposed  monitoring  panel.  Would  it  receive  summary  details  of  the  reports  of 
the  several  boards  of  scientific  counselors  or  the  reports  of  the  boards  in  their 
entirety?  The  subject  of  scope  was  also  considered:  14  boards  each  meeting 
twice  a  year  would  produce  28  major  reports,  each  a  synopsis  of  a  number  of  site 
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visits  during  which  the  actual  quality  assessment  occurs.  Would  the  DAC  in  its 
current  form  have  sufficient  resources  to  follow  the  intramural  programs  of  all 
the  institutes?  How  could  the  proposed  panel  truly  monitor  the  review  process 
without  adding  another  layer  to  the  process? 

Serious  concerns  also  were  raised  about  how  the  responsibilities  of  the  proposed 
DAC  panel  would  compare  with  the  legislatively  mandated  responsibilities  of 
the  institutes'  national  advisory  councils.  The  Panel  felt  that  quality  issues 
related  to  the  intramural  programs  were  well  within  the  purview  of  the  national 
advisory  councils'  legal  responsibilities.  Reports  of  reviews  completed  by  the 
boards  of  scientific  counselors  should  be  presented  to  the  councils,  which  have 
the  subsequent  authority  and  responsibility  to  advise  the  institute  directors. 

Finally,  the  Consultant  Panel  discussed  what  role,  if  any,  would  be  appropriate 
for  the  DAC  to  play  in  ensuring  that  the  intramural  review  procedures  continue 
to  meet  the  standards  of  excellence  envisioned  by  the  NAS/IOM  study  commit¬ 
tee.  The  Panel  concluded  that  while  the  DAC  should  not  have  a  continuing  role 
of  monitoring  the  review  process  as  recommended  in  the  NAS/IOM  report,  it 
should  reexamine  this  issue  in  2  years  and  periodically  thereafter  to  examine  the 
progress  made  in  strengthening  the  intramural  scientific  review  procedures. 

Evaluation  of  the  Scientific  Director 

The  Consultant  Panel  next  addressed  the  NAS/IOM  study  recommendation  that 
called  for  a  quadrennial  review  of  the  scientific  directors  and  their  intramural 
programs  by  an  external  group. 

The  NAS/IOM  study  committee  found  that  such  a  review  is  necessary  to  ensure 
the  vitality  of  the  intramural  program.  Under  the  leadership  of  the  institute 
director,  and  with  the  advice  of  the  board  of  scientific  counselors,  the  scientific 
director  exercises  a  decisive  role  in  influencing  the  intramural  program.  The 
NAS/IOM  report  noted,  "The  responsibility  of  the  scientific  director  requires 
that  he/ she  have  the  scientific  vision  needed  to  allocate  intramural  resources 
productively,  as  well  as  to  function  as  a  highly  skilled  manager."  The  NAS/IOM 
study  committee  also  recommended  that  scientific  directors  receive  additional 
compensation,  in  view  of  the  importance  of  their  role. 

The  Panel  discussion  initially  focused  on  whether  a  review  of  the  scientific 
quality  of  the  intramural  program  as  a  whole  would  serve  as  an  adequate  evalua¬ 
tion  of  the  scientific  director.  The  Panel  recognized  that  the  best  work  accom¬ 
plished  in  the  institutes  was  of  superb  quality  but  disagreed  with  the  idea  that 
program  quality  as  a  whole  could  serve  as  the  sole  measure  of  performance. 

There  was  extensive  discussion  about  the  dual  roles  performed  by  many  of  the 
scientific  directors;  namely,  maintaining  an  active  laboratory  conducting  scien¬ 
tific  research  and  providing  administrative  leadership  and  intellectual  stimulus 
for  the  intramural  program.  The  first  role  is  measured  by  the  quality  of  the 


18 


Report  of  the  Consultant  Panel 


individual's  scientific  accomplishments,  and  the  second  is  measured  by  the  scien¬ 
tific  quality  of  the  intramural  program  as  a  whole. 

The  Panel  felt  that  for  those  scientific  directors  directly  involved  in  continuing 
research,  there  should  be  a  strict  separation  of  the  scientific  evaluation  from  the 
administrative  evaluation  and  that  a  scientific  review  by  the  board  of  scientific 
counselors  should  occur  on  the  same  4-year  cycle  with  the  other  scientists  in  the 
program. 

With  regard  to  evaluating  the  administrative  duties,  the  Panel  recognized  that 
the  scientific  director  is  subject  to  an  annual  review  through  the  Federal  perform¬ 
ance  evaluation  system.  However,  the  Panel  strongly  felt  that  the  board  of  scien¬ 
tific  counselors,  as  an  external  body,  should  assess  how  well  the  intramural 
program  is  administered  and  what  effect  that  administration  has  on  the  overall 
quality  of  the  research  program. 

Issues  Related  to  Strengthening  the  Intramural 
Scientific  Review  Process 

In  the  course  of  its  deliberations,  the  Panel  addressed  several  additional  issues 
related  to  those  raised  by  the  NAS/IOM  report.  The  Panel  examined  the 
procedures  for  selecting  reviewers  and  the  mechanisms  for  ensuring  that  there  is 
an  appropriate  response  to  the  review.  The  Consultant  Panel  was  concerned 
about  negative  perceptions  of  the  review  procedures,  particularly  those  that 
called  into  question  the  accountability  and  independence  of  the  intramural 
review  process.  Among  the  issues  addressed  by  the  Panel,  with  the  goal  of 
strengthening  the  review  process,  were  the  following: 

•  How  should  counselors  be  selected,  and  what  are  the  appropriate  size  and 
length  of  service  for  the  boards  of  scientific  counselors? 

•  Is  there  a  need  for  a  set  of  minimum  codified  standards  that  would  apply 
across  all  the  institutes? 

•  What  is  the  nature  of  the  orientation  received  by  counselors? 

•  How  should  the  review  be  structured,  to  whom  should  the  report  of  the 
review  be  delivered,  and  what  should  be  the  mechanisms  for  following 
through? 

•  What  is  the  most  appropriate  mechanism  for  responding  to  the  report  of  the 
review  by  the  boards  of  scientific  counselors? 

Boards  of  Scientific  Counselors 

The  Panel  concurred  with  findings  in  the  NAS/IOM  report  that  the  nomination 
of  scientific  counselors  by  the  scientific  director  whose  program  is  to  be  reviewed 
contributed  to  perceptions  of  conflict  of  interest.  The  Panel  believed  the  nominat¬ 
ing  individual  should  be  the  chairperson  of  the  board  of  scientific  counselors 
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with  the  advice  of  the  scientific  director.  The  list  of  nominees  should  be  pro¬ 
vided  by  the  chairperson  to  the  director  of  the  institute  and  the  Deputy  Director 
for  Intramural  Research,  NIH,  for  approval.  The  appointment  authority,  with  the 
exception  of  NCI,  should  remain  with  the  Director,  NIH.  Ad  hoc  reviewers  used 
for  specific  site  visits  also  should  be  selected  by  the  board  chairperson  with  the 
scientific  director's  advice. 

Concern  was  expressed  by  the  Panel  that  in  a  4-year  term  of  office,  no  single 
scientific  counselor  would  have  the  opportunity  to  review  any  laboratory  more 
than  once.  The  Panel  believed  the  review  process  would  be  enhanced  if  a  portion 
of  the  board  had  some  prior  experience  reviewing  a  particular  laboratory  and  felt 
continuity  could  be  accomplished  by  lengthening  the  term  of  appointment  for 
scientific  counselors  to  5  years. 

It  also  was  thought  that  continuity  would  be  enhanced  by  increasing  the  size  of 
the  boards.  This  discussion  centered  on  the  most  appropriate  size  of  a  board,  the 
limited  number  of  experts  available  for  specific  disciplines,  the  difficulty  in 
recruiting  them,  and  the  NIH  restrictions  that  prohibit  individuals  from  serving 
on  more  than  one  board.  Despite  the  recruitment  difficulties,  the  panel  strongly 
believed  that  the  institutes  should,  to  the  extent  possible,  increase  the  number  of 
individuals  serving  on  their  boards  of  scientific  counselors. 

The  Panel  considered  how  board  members  could  make  a  meaningful  contribu¬ 
tion  to  the  review  process  at  the  end  of  their  appointments.  The  Panel  believed  it 
would  be  worthwhile  to  conduct  exit  surveys  of  the  board  members  on  issues  of 
program  quality  and  the  efficacy  of  the  review  process.  This  survey  was  envi¬ 
sioned  as  a  standard  form  that  the  departing  members  could  complete  after  their 
last  meeting  and  return  to  the  chairperson,  who  would  subsequently  distribute 
them  to  sitting  board  members. 

Review  Function  and  Process 

The  question  of  uniformity  of  review  procedures  among  the  institutes  was  the 
focus  of  a  concerted  discussion  by  the  Panel.  Although  they  acknowledged  that 
differences  exist  among  the  institutes  that  require  variations  in  procedures,  the 
Panel  members  felt  that  a  minimum  set  of  uniform  standards  should  be  devel¬ 
oped  and  codified.  The  guidance  found  in  Chapter  3005  together  with  the  recom¬ 
mendations  of  the  panel  for  strengthening  the  review  process  could  serve  as  a 
basis  for  these  standards. 

The  Panel  believed  that  the  Deputy  Director  for  Intramural  Research,  NIH, 
should  evaluate  the  institutes'  unique  procedures  to  ensure  that  the  uniform 
requirements  of  NIH  are  met  and  communicate  the  experiences  of  the  different 
processes  to  all  of  the  scientific  directors.  The  Deputy  Director  for  Intramural 
Research,  NIH,  should  also  publish  orientation  guidelines  for  the  scientific  coun¬ 
selors  that  describe  the  uniform  standards,  the  goals  of  the  review  process,  and 
the  responsibilities  of  the  counselors.  These  guidelines  should  be  provided  to 
counselors  as  part  of  their  briefing  materials. 
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Panel  members  deliberated  on  the  nature  of  the  review  and  the  areas  of  the 
program  for  which  advice  should  be  sought  from  the  boards.  The  Panel  believed 
that  the  boards  of  scientific  counselors  should  provide  advice  on  science,  specific 
projects  including  new  areas  of  development,  tenure  and  promotion  decisions, 
resource  allocation,  and  other  administrative  matters. 

The  followup  to  the  scientific  review  process  was  of  particular  interest  to  the 
Panel,  and  its  deliberations  focused  on  mechanisms  designed  to  facilitate  mean¬ 
ingful  exchange  between  boards  of  scientific  counselors  and  their  respective 
institutes'  intramural  programs.  Although  the  Panel  believed  that  the  board 
should  provide  an  oral  summary  of  its  review  before  adjourning,  the  consultants 
thought  it  imperative  that  the  briefing  be  attended  by  both  institute  director  and 
scientific  director,  as  well  as  by  the  Deputy  Director  for  Intramural  Research, 
NIH. 

The  Consultant  Panel  also  considered  the  impact  of  timing  during  the  review 
cycle  and  felt  that  within  a  month  of  adjournment,  the  board  should  provide  a 
final  written  report  of  its  review  to  the  scientific  director.  Within  6  months  of 
receiving  the  report,  but  before  the  next  board  meeting,  the  scientific  director 
should  respond  to  the  board's  report  indicating  any  areas  of  agreement  and  dis¬ 
agreement  and  the  changes  that  will  be  undertaken.  Also  at  this  time,  a  copy  of 
the  board  report  and  the  scientific  director's  response  to  it  should  be  sent  to  the 
institute  director;  the  national  advisory  council;  the  NIH  Deputy  Director  for 
Intramural  Research;  and  the  Director,  NIH. 

No  later  than  1  year  after  the  review,  a  written  response  should  be  provided  to 
the  board  of  scientific  counselors  by  the  scientific  director,  summarizing  the 
results  of  any  actions  taken. 

Finally,  the  Panel  clearly  reaffirmed  the  mandated  oversight  role  of  the  national 
advisory  councils.  The  Panel  concluded  that  the  institute's  advisory  council 
should  review  the  intramural  research  program  at  least  once  a  year  and  that  it 
should  be  provided,  at  a  minimum,  with  the  report  of  the  board  of  scientific 
counselors  and  the  written  response  of  the  scientific  director. 
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Summary  of  Recommendations 

Monitoring  of  the  Intramural  Research  Program 

1.  An  ad  hoc  Consultant  Panel  of  the  DAC  should  be  convened  in  2  years,  and 
periodically  thereafter,  to  review  progress  on  the  following  recommendations 
and  changes  in  the  intramural  scientific  review  process. 

Evaluation  of  the  Scientific  Director 

2.  The  scientific  and  administrative  performance  of  the  scientific  directors 
should  be  evaluated  every  4  years  by  the  boards  of  scientific  counselors.  The 
chairperson  of  the  board  of  scientific  counselors  should  provide  the  institute 
director  with  a  written  evaluation  of  the  scientific  director,  including  a  discus¬ 
sion  of  the  overall  direction  of  the  intramural  program  and  the  efficacy  of  the 
review  process.  The  institute  director  may  request  an  evaluation  of  the  scien¬ 
tific  director  by  the  chairperson  each  year. 

Boards  of  Scientific  Counselors 

3.  Nominations  for  the  board  of  scientific  counselors  should  be  provided  to  the 
director  of  the  institute  by  the  chairman  of  the  board  of  scientific  counselors 
with  the  advice  of  the  scientific  director.  The  institute  director  and  the 
Deputy  Director  for  Intramural  Research,  NIH,  should  provide  a  final  review 
and  approval  of  the  list  of  nominees.  The  appointment  authority,  with  the 
exception  of  NCI,  should  remain  with  the  Director,  NIH.  Ad  hoc  reviewers 
should  be  selected  by  the  chairperson  of  the  board  of  scientific  counselors 
with  the  advice  of  the  scientific  director. 

4.  The  term  of  office  for  scientific  counselors  should  be  increased  to  5  years  to 
facilitate  continuity  of  review  and  to  allow  the  review  of  specific  programs 
more  than  once  by  the  same  counselor(s). 

5.  The  institutes  should,  to  the  extent  possible,  increase  the  number  of  indi¬ 
viduals  serving  on  their  boards  of  scientific  counselors. 

6.  Upon  completion  of  their  appointment,  scientific  counselors  should  be  sur¬ 
veyed  by  the  chairperson  of  the  board  of  scientific  counselors  on  issues  of 
program  quality  and  the  scientific  review  process.  This  survey  information 
should  be  provided  to  current  board  members. 

Review  Function  and  Process 

7.  Uniform  standards  for  scientific  review  procedures  should  be  developed  and 
codified. 

8.  Within  certain  specific  requirements,  each  institute  should  be  allowed  to 
develop  its  own  procedures  for  the  organization  and  structure  of  the  scientific 
review  meetings.  The  Deputy  Director  for  Intramural  Research,  NIH,  should 
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evaluate  the  institutes'  procedures  to  ensure  that  the  uniform  requirements  of 
NIH  are  met  and  to  communicate  the  experiences  of  the  different  processes  to 
all  of  the  scientific  directors. 

9.  Scientific  counselors  should  be  provided  with  orientation  guidelines  that 
describe  the  review  procedures,  the  goals  of  the  process,  and  the  respon¬ 
sibilities  of  counselors.  These  guidelines  should  be  published  by  the  Deputy 
Director  of  Intramural  Research,  NIH. 

10.  The  boards  of  scientific  counselors  should  provide  evaluation  and  advice  on 
science,  specific  projects  (including  new  areas  of  development),  tenure  and 
promotion  decisions,  resource  allocation,  and  other  administrative  matters. 

11.  All  board  members  should  be  provided  with  a  summary  of  the  organization 
of  the  laboratory  to  be  reviewed;  the  various  sections;  all  personnel,  space 
usage,  operating  budget,  and  outside  contracts;  and  Cooperative  Research 
and  Development  Agreements.  Each  scientist  being  reviewed  should  supply 
details  of  ongoing  work,  a  curriculum  vitae,  reprints  of  the  three  to  five  most 
important  recent  publications,  and  an  analysis  of  the  amount  of  support  staff 
and  space  that  scientist  uses. 

12.  Before  the  board  adjourns,  an  oral  summary  of  the  review  should  be 
presented  to  the  scientific  director;  the  institute  director;  and  the  Deputy 
Director  for  Intramural  Research,  NIH;  or  their  designees. 

13.  Within  a  month  of  adjournment,  the  board  should  provide  a  written  report  of 
its  review  to  the  scientific  director.  Those  portions  of  the  report  that  pertain 
to  particular  laboratory/branch  chiefs  should  be  provided  to  them  by  the 
scientific  director.  The  scientific  director  and  the  laboratory/branch  chief 
should  discuss  the  evaluations  and  recommendations  in  the  report.  The  scien¬ 
tific  director  may  also  choose  to  discuss  appropriate  parts  of  the  report  with 
section  chiefs  and  individual  scientists  who  have  been  reviewed. 

14.  Within  6  months  of  receiving  the  board's  report,  but  before  the  next  board  of 
scientific  counselors'  meeting,  the  scientific  director  should  respond  to  the 
board  indicating  areas  of  agreement  and  disagreement  and  any  possible 
changes  that  may  be  undertaken.  A  copy  of  the  report  and  the  response  of 
the  scientific  director  should  be  sent  to  the  director  of  the  institute;  the  Deputy 
Director  for  Intramural  Research,  NIH;  the  Director,  NIH;  and  the  institute's 
national  advisory  council. 

15.  No  later  than  1  year  after  the  review,  a  written  response  should  be  provided 
to  the  board  of  scientific  counselors  by  the  scientific  director,  summarizing 
the  results  of  any  actions  taken. 

16.  The  institute's  national  advisory  council  should  be  provided  an  overview  of 
the  institute's  intramural  research  program  at  least  once  a  year.  The  overview 
should  include,  at  a  minimum,  a  discussion  of  the  board  of  scientific 
counselors'  report  and  the  response  of  the  scientific  director. 
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Conclusion 

This  study  and  the  Consultant  Panel's  recommendations  for  strengthening  the 
NIH  intramural  scientific  review  procedures  are  in  response  to  the  recent  review 
of  the  NIH  intramural  program  by  NAS/IOM. 

In  its  report,  the  NAS/IOM  study  committee  expressed  concerns  about  whether 
the  review  process  was  sufficient  to  sustain  the  future  excellence  of  the  intra¬ 
mural  program  at  NIH.  The  study  committee  felt  that  the  mechanics  of  the 
review  process  itself  were  sound,  but  it  framed  the  issue  in  terms  of  what 
changes  could  be  made  to  demonstrate  that  tough  questions  about  the  review 
process  were  being  asked.  Concerns  about  the  appointment  process  for  scientific 
counselors  and  the  procedures  for  following  through  on  reports  by  boards  of 
scientific  counselors  also  were  expressed  by  the  study  committee. 

The  NAS/IOM  solutions  to  these  issues  included  an  enhanced  monitoring  role 
for  the  DAC,  and  the  requirement  that  the  scientific  directors  and  their  intra¬ 
mural  programs  be  reviewed  by  an  external  body  every  4  years. 

The  Panel  concurs  with  the  NAS/IOM  study  committee's  findings  that  measures 
should  be  taken  to  strengthen  the  scientific  review  process  and  especially 
applauds  its  recognition  of  the  special  contributions  of  the  scientific  directors  to 
the  excellence  of  the  intramural  research  programs. 

Since  its  inception,  the  DAC  has  served  an  advisory  capacity  on  broad  science 
policy  matters  pertinent  to  the  NIH  mission,  and  the  Consultant  Panel  believes  it 
is  appropriate  for  the  DAC  to  examine  and  provide  its  advice  periodically  on  the 
scientific  review  procedures  of  the  intramural  research  program.  However,  the 
Panel  does  not  believe  that  the  DAC  is  the  most  appropriate  body  for  monitoring 
the  review  process  and  has  instead  sought  to  strengthen  the  existing  system. 

The  strengthening  of  the  process  should  occur  through  the  implementation  of  the 
Consultant  Panel's  recommendations.  The  recommendations  provide  specific 
means  for  strengthening  the  boards  of  scientific  counselors,  a  more  rigorous 
review  and  response  process  based  on  codified  standards,  and  a  more  impartial 
counselor  selection  process.  Finally,  the  Panel  recommends  procedures  for  the 
periodic  review  of  the  scientific  and  administrative  performance  of  the  scientific 
directors. 
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NOTE 

References  to  the  scientific  review  of  the  NIH  intramural  research  program  have 
been  excerpted  from  pages  4, 5, 8, 9,  41, 42, 43, 138  and  139  of  the  National 
Academy  of  Sciences,  Institute  of  Medicine  report,  A  Healthy  NIH  Intramural 
Program:  Structural  Change  or  Administrative  Remedies?  Copies  of  the  report  are 
available  from  the  Office  of  Intramural  Affairs,  NIH. 
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(From  NAS/IOM  Report,  pp,  4-5) 


Quality  of  the  Intramural  Program 

The  committee  believes  that  unless  the  quality  of  research  in  the 
intramural  program  is  excellent,  the  investment  of  the  government  is  not 
justified.  The  problems  of  measuring  quality  of  scientific  institutions  are 
well  known.  The  committee  used  several  indicators  of  quality,  such  as 
citation  analysis  and  a  review  of  notable  achievements.  It  also  examined 
quality  assurance  mechanisms.  These  indicators  suggest  that  no  serious 
decline  in  quality  has  occurred  in  one  of  our  nation's  most  important  centers 
of  biomedical  research. 

One  can  identify  many  investigators  who  are  among  the  most  respected  in 
their  fields.  However,  not  all  work  in  the  intramural  program  meets  the  same 
high  standards.  This  variability  perhaps  is  to  be  expected  in  any  research 
organization  of  the  size  and  scope  of  the  intramural  program.  The  committee 
was  unable  to  determine  the  extent  to  which  notable  scientists  mask  a  cadre  of 
less  productive  scientists.  Nevertheless,  it  is  the  committee's  judgment  that 
further  improvements  in  the  quality  of  the  program  are  essential  and 
attainable . 

There  has  been  long-standing  concern  in  the  biomedical  community  that  the 
review  process  for  the  intramural  research  program  lacks  the  rigor  of  the 
competitive  peer  review  process  of  the  extramural  program.  Although  in  recent 
years  NIH  has  taken  action  to  improve  this  process,  inadequacies  remain  in  the 
appointment  process  and  the  degree  to  which  recommendations  of  the  Board  of 
Scientific  Counselors  are  given  serious  consideration.  The  committee  believes 
it  is  particularly  important  that  accountability  to  a  disinterested  body, 
external  to  the  intramural  program  and  institutes,  has  oversight 
responsibility  to  ensure  the  integrity  of  the  review  process.  The  committee 
does  not  recommend  that  the  intramural  program  adopt  the  procedures  by  which 
the  extramural  competitive  grants  are  evaluated.  But,  a  more  credible  and 
independent  peer  review  system- -  suitable  for  the  environment- -  is  essential  to 
sustain  the  future  vitality  of  the  intramural  program.  This  is  a  key  step  in 
ensuring  the  most  effective  use  of  the  resources  invested  in  the  program. 

A  rigorous  review  process  is  necessary  but  not  sufficient  to  sustain 
quality.  Under  the  leadership  of  the  institute  director,  the  scientific 
director  of  each  institute  is  key  to  the  success  of  the  intramural  program, 
providing  both  intellectual  and  administrative  leadership.  Not  only  do 
scientific  directors  control  resources,  but,  less  tangibly,  they  are 
responsible  for  the  spirit  and  morale  of  the  institute. 

The  committee  believes  that  the  qualities  of  demonstrated  scientific 
achievement,  leadership,  and  administrative  ability  that  are  needed  for  this 
position  are  rare  commodities.  To  attract  people  of  sufficient  stature 
requires  that  a  premium  be  paid. 
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(From  pp .  8-9) 


Improving  the  Review  of  the  Intramural  Program 

Several  of  the  committee's  recommendations  are  designed  to  maintain  high 
scientific  standards.  Recommendations  to  give  NIH  managers  the  necessary 
flexibility  to  compete  for  personnel  and  provide  a  productive  work  environment 
are  clearly  intended  to  enhance  the  intellectual  capital  of  the  program.  The 
committee  believes,  however,  that  disinterested  review  of  the  intramural 
research  programs  and  assurance  of  implementation  of  reasonable 
recommendations  also  is  essential  to  credible  quality  assurance. 

Two  recommendations  address  the  review  and  resource  allocation  process. 

A  panel  chaired  by  a  member  of  the  NIH  Director's  Advisory  Committee  should  be 

established  to  monitor  the  intramural  research  program  review.  The  functions 
of  this  panel  would  be  to  oversee  the  integrity  of  the  process,  while  taking 
care  not  to  replicate  the  activities  of  the  Boards  of  Scientific  Counselors. 
Rather,  its  oversight  should  focus  on  areas  that  are  most  vulnerable  to 
criticism,  namely  the  selection  of  the  reviewers  and  the  appropriate  response 
to  recommendations. 

Each  of  the  scientific  directors  and  their  intramural  programs  should  be 

reviewed  as  a  whole  every  four  years  by  an  external  group.  The  review  report 

should  be  submitted  to  the  director  of  the  relevant  institute,  the  NIH  Deputy 

Director  for  Intramural  Research,  the  Director  of  NIH,  and  the  Director's 

Advisory  Committee.  The  committee  believes  such  a  review  to  be  necessary 
because  of  the  importance  of  ensuring  the  vitality  of  the  intramural  program. 
The  intent  of  the  periodic  review  is  not  to  limit  arbitrarily  the  term  of  the 
scientific  director,  but  rather  to  put  in  place  a  process  that  will  ensure 
vigorous  leadership.  The  responsibility  of  the  scientific  director  requires 
having  the  scientific  vision  needed  to  allocate  intramural  resources 
productively,  as  well  as  function  as  a  highly  skilled  manager.  To  recruit  and 
retain  scientists  with  this  extraordinary  set  of  attributes,  the  committee 
recommends  that  those  holding  the  position  of  scientific  director  receive 

additional  compensation.  This  will  become  possible  under  the  recommended 
personnel  demonstration  program. 

(From  pp .  41-43) 

Review  of  Intramural  Laboratories 

NIH  policy  states  that  "all  research  conducted  intramurally  must  be 
reviewed  regularly  by  highly  qualified  outside  scientists."  For  this  purpose, 
each  institute  appoints  a  board  of  scientific  counselors  to  review  the 
intramural  work  of  each  institute.  The  boards  are  composed  of  scientists  with 
outstanding  achievements  and  expertise  in  the  fields  under  review.  The 
institute's  scientific  director  may  invite  additional  experts  to  supplement 
the  expertise  of  the  board  members  for  specific  reviews. 

Nominations  for  board  membership  are  made  by  scientific  directors  and  are 
approved  by  the  Deputy  Director  of  NIH  and  the  individual  institute  director. 
Members  are  appointed  by  the  Director  of  NIH.  During  these  reviews,  the  board 
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of  scientific  counselors  considers  the  quality  of  research  accomplished  since 
the  last  review  and  its  contribution  to  the  institute's  mission.  Over  a 
period  of  time,  each  laboratory  and  tenured  principal  investigator  is 
examined,  as  are  scientists  being  considered  for  tenured  positions.  The  board 
offers  advice  to  the  scientific  director  regarding  allocation  of  personnel 
positions,  funding  of  specific  research  areas,  and  future  directions  for 
research . 

Prior  to  the  review,  members  of  the  board  of  scientific  counselors 
receive  written  descriptions  of  the  laboratories'  research,  staff 
qualifications,  budget  summaries,  space  allocations,  and  research  support 
contracts.  When  the  board  meets,  tenured  scientists  and  junior  staff  report 
on  present  and  planned  research- -board  members  have  an  opportunity  to  question 
the  scientists  and  to  visit  laboratories. 

Within  four  months  of  review,  a  report  of  the  board's  findings  and 
recommendations  is  submitted  for  information  to  the  scientific  director,  the 
director  of  the  institute,  the  NIH  Deputy  Director  for  Intramural  Research, 
and  the  Director  of  NIH.  Following  review  by  the  boards  of  scientific 
directors  of  all  the  institutes,  the  report  is  sent  to  each  institute's 
national  advisory  council  for  information  and  comment.  The  scientific 
director  is  required  to  report  to  the  board  of  scientific  counselors,  at  the 
earliest  practical  date,  on  actions  taken  on  their  recommendations  (Eberhart, 
1982;  National  Institutes  of  Health,  1986a). 

Feedback  to  the  board  of  scientific  counselors  was  of  particular  concern 
to  members  of  the  1984  Institute  of  Medicine's  Panel  to  Study  the  Current 
Organizational  Structure  of  the  NIH  (Institute  of  Medicine,  1984),  who 
interviewed  members  of  the  Boards.  Many  board  members  expressed  a  desire  to 
be  better  informed  regarding  the  institutes'  implementation  of  their 
recommendations,  suggesting  that  their  recommendations  were  not  always  given 
serious  consideration. 

In  1985,  the  Director  of  NIH  recruited  a  group  of  external  reviewers  to 
examine  the  management  of  the  Clinical  Center.  As  part  of  the  review,  8 
experts  from  a  broad  spectrum  of  clinical  disciplines  evaluated  50  current  or 
recently  completed  protocols  selected  by  the  director  and  deputy  director  of 
the  Clinical  Center.  These  protocols  were  chosen  to  represent  "high  quality" 
examples  of  research  that  made  intensive  use  of  the  Clinical  Center  resources. 
The  reviewers  concluded: 

There  was  substantial  variation  in  the  quality  of  the 
protocols  reviewed,  from  truly  outstanding  to  quite  poor, 
and  there  was  also  considerable  variation  of  quality  in 
and  among  the  Institutes.... 

The  reviewers  noted  differences  in  the  scientific  merit 
review  mechanisms  among  the  Institutes  and  commented  on 
the  need  for  a  more  rigorous  review  mechanism  in  those 
Institutes  where  the  protocols  were  weak  (National 
Institutes  of  Health,  1985). 
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At  about  this  time,  Congress  became  concerned  about  the  intramural  review 
process  in  the  context  of  deliberating  on  the  1985  Health  Research  Extension 
Act.  The  conference  committee  report  notes  the  following: 

The  conference  agreement  requires  the  Director  of  NIH  to 
establish  procedures  for  periodic  technical  and  scientific 
peer  review  of  all  intramural  research  conducted  at  the 
National  Institutes  of  Health.  It  is  not  the  conferees' 
intent  that  the  review  procedures  for  intramural  research 
be  the  same  as  those  for  extramural  research.... 

An  entity  conducting  peer  review  of  intramural  research  is 
to  provide  the  institute's  advisory  council  with  a  written 
description  of  the  research,  the  results  of  the  review  and 
the  recommendations  of  the  reviewing  entity.  The 
conference  agreement  authorizes,  but  does  not  require,  the 
advisory  council  to  make  recommendations  to  the  institute 
director  regarding  intramural  activities  conducted  by  the 
institute  (Health  Research  Extension  Act,  1985). 

As  a  result  of  congressional  concern,  the  review  process  has  been 
tightened.  The  recommendations  of  the  boards  of  scientific  counselors  must  be 
answered  in  writing  by  the  scientific  director  of  the  institute  at  the  next 
meeting  of  the  board  (National  Institutes  of  Health,  1986b). 

The  committee  commends  NIH  for  its  actions  to  strengthen  the  review 
procedures.  It  has,  however,  identified  two  specific  points  in  the  review 
system  that  lack  mechanisms  to  ensure  objectivity  and  accountability.  First, 
while  few  question  the  stature  of  those  who  serve  on  boards  of  scientific 
counselors  and  as  ad  hoc  consultants,  the  fact  that  they  are  nominated  by  the 
scientific  director  of  the  institute  whose  program  will  be  under  review 
compromises  the  external  credibility  of  their  report.  Second,  despite 
improved  feedback  to  the  board  of  scientific  counselors  on  the  implementation 
of  their  recommendations,  the  scientific  director  is  at  no  time  accountable  to 
anyone  outside  the  institute.  Because  this  process  is  so  often  unfavorably 
compared  with  the  rigorous  extramural  peer  review- -particularly  by  some  who 
believe  that  intramural  funds  would  be  better  spent  for  unfunded  extramural 
projects- -it  is  important  that  the  system  has  real  and  visible  safeguards. 

A  rigorous  review  process  is  necessary  but  not  sufficient  to  sustain 
quality.  The  scientific  director  of  each  institute  is  key  to  the  success  of 
the  research  program,  providing  both  intellectual  and  administrative 
leadership.  Not  only  do  the  scientific  directors  control  resources,  but,  less 
tangibly,  they  are  responsible  for  the  scientific  esprit  of  the  institute. 
Subsequent  chapters  include  discussion  of  the  problems  of  recruiting  and 
retaining  outstanding  individuals  for  these  jobs. 


A-  8 


(From  pp.  138-139) 


Assuring  Quality.  The  committee  has  four  major  recommendations  designed 
to  assure  that  the  quality  of  the  intramural  program  be  maintained  at  a  high 
level  for  the  future.  The  first  three  relate  to  assuring  that  the  review 
process  can  be  seen  to  be  rigorous  and  leads  to  wise  use  of  intramural 
resources..  The  last  recommends  a  program  to  make  NIH  competitive  for  a  share 
of  the  most  promising  young  investigators  in  the  country. 

The  Review  Process.  To  improve  the  external  validity  of  the  review 
process,  the  committee  recommends  that  a  panel  chaired  by  a  member  of  the 
Director's  Advisory  Committee  should  be  established  to  monitor  the  intramural 

research  program  review.  The  functions  of  this  panel  would  be  to  monitor  the 
integrity  of  the  process,  while  taking  care  not  to  replicate  the  activities  of 
the  boards  of  scientific  counselors.  Its  oversight  should  focus  on  areas  that 
are  most  vulnerable  to  criticism,  namely  the  selection  of  the  reviewers  and 
the  implementation  of  recommendations. 

Each  of  the  scientific  directors  and  their  intramural  programs  should  be 

reviewed  as  a  whole  every  four  years  by  an  external  group.  The  review  report 

should  be  submitted  to  the  director  of  the  relevant  institute,  the  NIH  Deputy 

Director  for  Intramural  Research,  the  Director  of  NIH.  and  the  Director's 

Advisory  Committee.  The  committee  believes  that  such  a  review  is  necessary 
because  of  the  importance  of  ensuring  the  vitality  of  the  intramural  program. 
The  intent  of  the  periodic  review  is  not  to  limit  arbitrarily  the  term  of  the 
scientific  director,  but  rather,  to  put  in  place  a  process  that  will  ensure 
vigorous  leadership.  Under  the  general  leadership  of  the  institute  director, 
and  with  the  advice  of  the  Board  of  Scientific  Counselors,  the  scientific 
director  exercises  a  decisive  role  in  influencing  the  intramural  program.  The 
right  person  in  this  position  functions  not  merely  as  a  caretaker,  but  a  force 
for  excellence.  The  responsibility  of  the  scientific  director  requires  that 
he/she  have  the  scientific  vision  needed  to  allocate  intramural  resources 
productively,  as  well  as  to  function  as  a  highly  skilled  manager.  Therefore, 
the  committee  recommends  that  those  holding  the  position  of  scientific 
director  receive  additional  compensation.  This  will  become  possible  under  the 
recommended  personnel  demonstration  program. 
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Agenda 

Strengthening  the  Scientific  Review 
Procedures  of  the  NIH  Intramural  Research  Program 

Tuesday,  September  5, 1989 

Building  31,  Conference  Room  9 
National  Institutes  of  Health 
Bethesda,  Maryland 


8:30  Opening  Remarks  . Dr.  Hill 

8:40  Statement  of  the  Issue . Dr.  Petersdorf 

Discussion 

9:10  Summary  Overview  of  Current  NIH  Intramural 

Scientific  Review  Procedures . Dr.  Rail 

9:30  Current  Intramural  Scientific  Review  Procedures 
at  the  National  Institute  on  Allergy  and 

Infectious  Diseases . Dr.  Gallin 

9:50  Current  Intramural  Scientific  Review  Procedures 

at  the  National  Institute  of  Child  Health  and  Human 

Development . Dr.  Levine 

10:10  Current  Intramural  Scientific  Review  Procedures 

at  the  National  Cancer  Institute . Dr.  Rabson 

10:30  Break 

10:45  Working  Paper  on  Proposed  NIH  Intramural 

Scientific  Review  Procedures . Dr.  Korn 

Discussion 

12:15  Lunch 
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1:15 

Discussion,  Report  Formulation  and 
Recommendations  Session  . 

. Consultant  Panel 

3:00 

Break 

3:15 

Resume  Panel  Session 

4:45 

Closing  Remarks . 

. Dr.  Hill 

5:00 

Adjournment 
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A.  Eu rpose  This  chapter  establishes  policy  and  procedures  for  outside 
scientific  review  and  evaluation  of  Intramural  research  at  NIH  bv 
Boards  of  Scientific  Counselors. 

B.  Background  Intramural  research  at  NIH  has  been  reviewed  by 
committees  of  scientists  from  outside  the  NIH  since  1956.  In  that 
year  the  first  committees*  called  Boards  of  Scientific  Counselors* 
were  appointed  to  review  intramural  research  In  the  Individual 
Institutes.  Board  procedures  have  evolved  over  the  years.  On 
March  4*  1985,  the  original  version  of  this  Manual  Chapter  was 
Issued  to  establish  policy  and  procedures  for  the  evaluation  of 
Intramural  research  by  the  Boards.  Subsequently,  on  November  20, 
1985,  Public  Law  99-158,  the  Health  Research  Extension  Act  of  1985 
was  enacted.  Including  a  provision  making  mandatory  what  was  then 
standard  policy  on  review  of  Intramural  research.  This  present 
chapter  Is  a  revision  of  the  March  4,  1985  Chapter;  It  makes 
explicit  the  present  law  concerning  Intramural  research  review.  It 
repeats  the  provisions  of  the  earlier  Chapter,  none  of  which  was 
changed  by  Public  Law  99-158. 

C.  References 

1.  Section  301  of  the  PHS  Act  authorizes  the  conduct  of  research. 
Investigations,  experiments,  demonstrations  and  studies 
relating  to  diseases  and  Impairments  of  man. 

2.  Sections  402(b)(3),  406(a)(3)  and  492(b)  of  the  Public  Health 
Service  Act,  as  amended  by  Section  2  of  Public  Law  99-158. 

3.  Memorandum  of  July  18,  1980,  from  Dr.  Robert  Goldberger, 

Deputy  Director  for  Science,  entitled  "Modus  Operandl  of 
Boards  of  Scientific  Counselors." 

4.  Report  by  Dr.  John  C.  Eberhart,  dated  November  3,  1982, 
entitled  "Review  and  Evaluation  of  Intramural  Research." 

5*  NIH  Public  Advisory  Groups.  October  1985. 

6*  The  NIH.  Committee  Management  Handbook  contains  Instructions 
and  sample  notices  concerning  the  meetings  of  Boards  of 
Scientific  Counselors.  (To  be  Issued.) 

D.  Definitions 

!•  Boards  of  Scientific  Cour.solnrs- 

Public  advisory  committees,  established  under  provisions  of 
the  Federal  Advisory  Committee  Act  (Public  Law  92-463)  which 
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review,  discuss,  and  evaluate  research  projects  and  programs 
conducted  by  the  NIH,  and  by  ADAMHA  Intramural  programs 
located  on  the  NIH  campus.  The  reviews  Include  evaluation  of 
the  work  of  Individual  scientists  as  well  as  of  the  quality  of 
the  research  Itself. 

2.  Qut&tde.  review. 

The  review  of  NIH  research  and  of  staff  scientists  by  highly 
qualified  scientists  from  outside  the  NIH  who  are  formally 
appointed  to  a  public  advisory  committee  called  a  Board  of 
Scientific  Counselors.  This  process  Is  referred  to  In  the  law 
as  "technical  and  scientific  peer  review." 

E.  Pol  Icy  It  Is  the  policy  of  the  National  Institutes  of  Health  (and 
of  ADAMHA  Institutes  with  Intramural  components  at  NIH)  that  all 
research  conducted  Intramurally  must  be  reviewed  regularly  by 
highly  qualified  outside  scientists.  The  review  will  Include  the 
quality  of  the  research  and  the  accomplishments  of  Individual 
scientists.  Scientists  within  Institutes  will  be  reviewed  by  their 
established  Boards  of  Scientific  Counselors.  Scientists  within 
service  Divisions  (Clinical  Center,  Division  of  Computer  Research 
and  Technology,  Division  of  Research  Services)  will  be  reviewed 
either  by  the  Board  of  Scientific  Counselors  of  a  collaborating 
Investigator  or  by  an  ad  hoc  committee  consisting  of  outside 
reviewers  and  Institute  scientists.  The  composition  of  a  given  ad 
hoc  committee  will  be  determined  by  the  Division  Director  In 
consultation  with  the  Deputy  Director  for  Intramural  Research. 

These  Committees  will  function  as  Boards  of  Scientific  Counselors 
and  will  follow  the  same  procedures  In  carrying  out  their  reviews. 

This  policy  Is  reinforced  by  Section  492(b)  of  the  PHS  Act: 

"The  Director  of  NIH  shall  establish  procedures  for  periodic 
technical  and  scientific  peer  review  of  research  at  the 
National  Institutes  of  Health.  Such  procedures  shall  require 
that-- 

(1)  the  reviewing  entity  be  provided  a  written  description  of 
the  research  to  be  reviewed,  and 

(2)  the  reviewing  entity  provide  the  advisory  council  of  the 
national  research  Institute  Involved  with  such 
description  and  the  results  of  the  review  by  the  entity, 

and  shall  authorize  such  review  to  be  conducted  by  groups 
appointed  under  sections  402(b)(6)  and  405(c)(3)." 
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These  reviews  and  any  resulting  recommendations  should  be  Included 
In  a  written  report  submitted  to  the  Scientific  Director#  who  will 
transmit  It  to  the  BID  Director#  the  Deputy  Director  for  Intramural 
Research#  and  the  Director#  NIH.  In  addition#  as  required  by  PHS 
Act  Section  492(b)  above#  the  Institute  will  provide  to  Its 
National  Advisory  Council  written  descriptions  of  the  research 
reviewed  by  the  Board  of  Scientific  Counselors#  and  the  results  of 
that  review.  Such  communications  should  be  made  to  the  Council  at 
least  annually.  As  stated  In  PHS  Act  Section  406(a) (3) (A) (1 )#  the 
Advisory  Council  ’’may  on  the  basis  of  the  materials  provided  under 
Section  492(b)(2)  respecting  research  conducted  at  the  Institute# 
make  recommendations  to  the  Director  of  the  Institute  respecting 
such  research.” 


F.  EcocMuma 

1.  Composition  of  Boards  Every  effort  shall  be  made  to  maintain 
the  full  complement  of  Board  members.  A  primary  consideration 
of  scientific  qualification  for  Board  membership  shall  be 
International  recognition  as  an  authority  In  the  fields  under 
review.  Each  Board  should  exhibit  reasonable  balance  In 
membership  with  respect  to  gender#  race#  and  geographical 
distribution  of  members’  Institutions.  A  board  may  make  use 
of  jid  iiQL  consultants  when  the  Scientific  Director  deems  It 
necessary.  Board  members  are  nominated  by  the  BID  Scientific 
Director  and#  with  the  exception  of  the  NCI#  approved  by  the 
BID  Director  and  the  Deputy  Director  for  Intramural  Research# 
and  appointed  by  the  Director#  NIH.  Under  the  PHS  Act#  Sec. 
413(b)(7)#  the  Director#  NCI#  has  authority#  In  consultation 
with  the  National  Cancer  Advisory  Board#  to  appoint  members  of 
NCI  Boards  of  Scientific  Counselors. 


2.  ExaqM£D£y...0.f Review. Meetiags  The  Boards  of  Scientific 
Counselors  shall  meet  often  enough  (ordinarily  two  or  three 
times  each  year)  to  assure  that  the  work  of  each  Intramural 
scientist  on  a  permanent  appointment  In  each  Laboratory  or 
Branch  Is  reviewed  at  least  once  every  four  years. 

3.  Information  Supplied  to  Boards  of  Scientific  Counselors  Prior 

to -Meeting  The  following  Information  shall  be  provided  to 
each  Board  reviewer  In  advance  of  the  meeting. 


For  each  Laboratory/Branch  being  reviewed; 


(a)  A  staff  list  of  all  employees  to  Include  name# 

occupational  specialty#  type  of  appointment  and  grade. 
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(b)  A  list  of  all  contracts  which  support  the  work  of  the 
Laboratory/Branch. 

For  each  scientist  whose  work  Is  to  be  reviewed: 

(c)  A  current  CV  and  Bibliography  and  recent  relevant 
reprints. 

(d)  A  succinct  summary  of  current  research  and  statement  of 
future  plans. 

(e)  The  amount  of  technical  and  postdoctoral  support. 

A  copy  of  the  last  Board  of  Scientific  Counselors  review  of 
the  Laboratory/Branch  should  be  available  to  the  Board.  The 
Conference  Report  for  this  section  of  the  PHS  Act  notes  that 
"the  written  description  should  present  the  overall  past 
accomplishments  of  the  laboratory  and  Its  Investigators  since 
the  last  review  was  conducted.  The  written  description  might 
also  present:  the  general  alms,  objectives  and  projected 
directions  of  the  research  projects  to  be  conducted; 
appropriate  background  Information  on  possible  future  research 
projects;  a  brief  summary  of  the  general  research  plan;  and  an 
Indication  of  the  Importance  of  the  research  projects." 

4.  Who  Is  to  be  Reviewed  Every  Independent  Intramural  scientist 
on  a  permanent  appointment  must  be  reviewed  and  evaluated.  In 
general,  advice  of  the  Board  of  Scientific  Counselors  also 
should  be  obtained  concerning  scientists  who  are  being 
considered  for  conversion  to  tenure.  The  Scientific  Director 
may.  In  addition,  choose  to  ask  the  Board  to  review  the  work 
of  other  staff  scientists.  Over  time,  the  active  program  of 
each  laboratory  or  other  Intramural  research  unit  Is  to  be 
reviewed. 

5.  Review  Format  Organization  and  structure  of  meetings  of  the 
Board  of  Scientific  Counselors  will  be  left  to  the  discretion 
of  the  Scientific  Director.  Review  formats  found  to  be  useful 
In  the  past  Include:  (a)  oral  presentations  by  scientists  to 
a  plenary  session  of  the  Board,  (b)  Individual  Interviews  of 
scientists  by  Board  members  or  consultants,  and  (c)  small 
group  Interviews  by  one  or  more  Board  members  or  consultants 
with  members  of  a  Section,  Unit  or  working  group.  The 
reviewers  should  be  given  an  opportunity  to  visit  the 
facilities  where  the  research  Is  carried  out.  If  they  wish. 
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6.  Reporting  of  Results  of  Reviews 

(a)  The  report  of  the  Board  of  Scientific  Counselors  Is  to  be 
a  narrative  critique  from  the  Chair  of  the  Board# 
following  the  outline  preferred  by  the  Scientific 
Director. 

(b)  Evaluations  of  Individual  Investigators  must  address#  at 
a  minimum#  the  following  questions:  are  good  questions 
being  asked#  are  appropriate  approaches  being  used  to 
obtain  answers#  and  are  the  resources  available  to  this 
scientist  appropriate  to  the  accomplishments  to  date? 
These  evaluations  should  be  written  by  members  of  the 
Board  of  Scientific  Counselors  or  consultants#  and  should 
reflect  the  collective  views  of  the  total  Board  (or  at 
least  a  majority). 

(c)  The  total  report  Is  to  be  sent  within  four  months  of  the 
review  to  the  Scientific  Director#  who  Is  responsible  for 
transmitting  It  to  the  BID  Director#  the  Deputy  Director 
for  Intramural  Research#  and  the  Director#  NIH.  The 
Office  of  Intramural  Affairs  will  distribute  copies  to 
the  Board  of  Scientific  Directors  and  schedule  Its 
discussion  by  all  members  of  that  group. 

(d)  At  least  annually  each  Institute  will  transmit  to  Its 
national  advisory  council  descriptions  of  the  research 
that  has  been  reviewed  by  Its  Board(s)  of  Scientific 
Counselors  during  the  preceding  year#  together  with  the 
results  of  the  reviews.  Consideration  shall  be  given  to 
providing  the  Items  listed  under  section  F3  In  this 
Chapter.  The  results  of  the  reviews  and  Board 
recommendations  may  be  communicated  to  the  Council 
orally#  by  the  Scientific  Director  or  the  Chair  of  the 
Board  of  Scientific  Counselors#  or  In  writing.  If  In 
writing#  the  evaluations  of  Individual  scientists  may# 
for  privacy  reasons#  be  omitted. 

7.  Followup  It  Is  the  responsibility  of  the  Scientific  Director 
to  report  back  to  the  Board  of  Scientific  Counselors  the 
action  taken  In  response  to  recommendations  made  by  the  Board. 
This  report  shall  describe  actions  taken  or  explain  why  It  was 
found  Impossible  or  Inadvisable  to  follow  the  recommendations. 

8.  Schedule  of  Reviews  Each  Scientific  Director  must  submit  to 
the  Deputy  Director  for  Intramural  Research  a  schedule  of  the 
proposed  dates  of  review  of  each  Laboratory/Branch  In  the 

NIH  MANUAL  RELEASE 

8/4/86 


D-7 


SUPERSEDES 

3/4/85 


3005 


Page  6 

3005  REVIEW  AND  EVALUATION  OF  INTRAMURAL  RESEARCH 
BY  BOARDS  OF  SCIENTIFIC  COUNSELORS 


Institute  or  Division  covering  the  next  four-year  review 
cycle.  This  schedule  must  be  updated  annually  and  submitted 
to  the  Deputy  Director  for  Intramural  Research  by  September  1 
of  each  year. 

G.  Additional  Information  For  more  Information  on  this  chapter, 
contact  the  Office  of  Intramural  Affairs  on  496-3561. 

H.  Additional  Copies  of  this  Chapter  For  extra  copies  of  this  chapter 
send  a  Form  NIH  414-5  to  the  Printing  and  Reproduction  Branch, 
Division  of  Technical  Services,  Building  31,  Room  3C02. 
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WORKING  PAPER  FOR  THE  NIH  DIRECTOR'S  SUBCOMMITTEE 
EVALUATING  THE  REVIEW  PROCESS  OF  NIH  INTRAMURAL  PROGRAMS 

I.  INTRODUCTION 

The  Director,  NIH  has  strongly  endorsed  the  emphasis  given  in  the  recent  IOM 
Report  on  the  NIH  Intramural  Program  for  (a)  a  sound  external  review  of 
intramural  NIH  research,  (b)  the  responsiveness  of  intramural  NIH  to  such 
reviews,  and  (c)  review  of  the  performance  of  the  Scientific  Directors.  To 
assure  these  goals  are  met,  the  ad  hoc  subcommittee  was  established  by  the 
Director,  NIH  to  evaluate  the  existing  procedures  for  review  of  the  Intramural 
Programs  and  to  recommend  procedures  for  (a)  periodic  review  of  the  Scientific 
Directors,  (b)  mechanisms  for  selection  of  the  members  of  the  Boards  of 
Scientific  Counselors,  and  (c)  implementation  and  follow-up  to  the  Boards' 
recommendations.  This  working  paper,  prepared  by  representatives  of  the 
Intramural  Community  of  NIH,  provides  a  set  of  recommendations  which  modifies 
the  current  procedures  to  establish  a  uniform,  rigorous  review  process  of  all 
of  the  Intramural  Programs  that  will  reinforce  the  existing  high  standards  of 
excellence  while  preserving  the  unique  features  of  the  various  Intramural 
Programs. 

II.  INTRAMURAL  NIH  AND  THE  ROLE  OF  THE  BOARDS  OF  SCIENTIFIC  COUNSELORS 

The  unique  aspects  of  Intramural  NIH  derive,  on  the  one  hand,  from  its 
unprecedented  stability  of  its  research  funding  and  support  infrastructure,  and, 
on  the  other  hand,  from  the  flexibility  that  derives  from  its  administrative 
structure.  The  stability  of  funding  has  produced  a  large  and  highly  varied 
complex  of  research  programs,  encompassing  all  of  the  biomedical  disciplines, 
that  can  undertake  long-term,  high-risk  investigations.  The  flexibility  of 
management,  together  with  the  scope  and  excellence  of  its  scientific  programs, 
allows  Intramural  NIH  to  respond  to  national  health  emergencies  with  speed  and 
efficiency,  as  exemplified  by  the  discovery  of  the  etiologic  agent  for  AIDS  and 
development  of  AZT  for  its  treatment. 

General  management  of  each  Intramural  Program  is  by  a  Scientific  Director,  who 
is  responsible  to  the  Institute  Director  and  who  also  interacts  closely  with 
the  Deputy  Director  for  Intramural  Research,  NIH,  and  the  Scientific  Directors 
of  the  other  Institutes.  Each  Intramural  Program  comprises  a  number  of 
Laboratories  and  Branches,  each  with  a  responsible  Chief,  and  most  Laboratories 
and  Branches  are  sub-divided  with  Sections,  each  with  a  responsible  Head  and  one 
or  more  independent  investigators  who  direct  the  research  of  post-doctoral 
fellows  and  support  staff. 

While  the  Institute  Director  is  ultimately  responsible  for  the  quality  of  the 
Intramural  Program,  the  Scientific  Director  has  the  direct  and  immediate 
managerial  and  program  responsibilities  and  also  serves  an  important  advisory 
role  to  the  Institute  Director  on  matters  of  science  policy.  The  Scientific 
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Director  must  direct  the  scientific  program  within  the  constraints  of  the  Civil 
Service  and  Public  Health  Service  personnel  systems,  and  with  relatively  fixed 
allotments  of  space,  personnel  and  budget.  Although  the  relative  constancy  of 
resources  establishes  a  desirable  stability,  the  restrictions  on  resources  also 
demand  a  high  degree  of  creativity  of  the  Scientific  Director  to  foster  exciting, 
innovative  research  programs  within  the  federal  system. 

The  performances  of  the  independent  investigators  are  subject  to  virtually  daily 
review  by  the  Laboratory/Branch  Chiefs  and  both  the  independent  investigators 
and  the  Laboratory/Branch  Chiefs  are  under  continual  review  by  the  Scientific 
Director  who  meets  frequently  with  the  Institute  Director.  In  addition,  all 
conversions  to  tenure  and  promotions  of  the  scientific  staff  within  any  one 
Institute  must  be  approved  by  the  NIH  Board  of  Scientific  Directors  of  all  of 
the  Institutes.  The  latter  provides  a  measure  of  external  peer  evaluation  of 
the  Intramural  Program  of  each  Institute  by  the  Scientific  Directors  of  the  other 
Institutes  and  the  Deputy  Director  for  Intramural  Research,  NIH,  who  chairs  the 
NIH  Board  of  Scientific  Directors.  Nonetheless,  despite  this  extensive  internal 
review  of  each  Intramural  Program,  it  is  essential  that  the  Intramural  Programs 
and  their  Scientific  Directors  receive  regularly  scheduled,  formal  scientific 
review,  evaluation  and  advice  by  external  Boards  of  Scientific  Counselors. 

The  first  important  role  of  the  Board  of  Scientific  Counselors  is  to  advise  the 
Scientific  Director  through  independent,  critical  reviews  and  assessments  of  the 
research  objectives  and  accomplishments  of  the  Laboratories  and  Branches  and  each 
of  the  independent  investigators.  The  Board  of  Scientific  Counselors  has  the 
responsibility  of  recommending  both  programmatic  changes  and  changes  in  level 
of  support  for  individual  scientists,  including  conversions  to  tenure  status  and 
promotions.  The  Scientific  Director  must  be  responsive  to  the  Board  of 
Scientific  Counselors  by  either  implementing  its  recommendations  or  providing 
the  reasons  for  not  doing  so. 

The  second  important  role  of  the  Board  of  Scientific  Counselors  is  to  review 
and  assess  the  performance  of  the  Scientific  Director.  In  large  measure,  the 
sum  of  the  reviews  of  the  Laboratories  and  Branches  will  provide  a.  continuing 
review  of  the  performance  of  the  Scientific  Director.  In  addition,  at  periodic 
intervals  the  Scientific  Director  will  be  specifically  reviewed  and  a  formal 
report  made  to  the  Institute  Director. 

III.  COMPOSITION  AND  SELECTION  OF  THE  BOARDS  OF  SCIENTIFIC  COUNSELORS 

The  Board  of  Scientific  Counselors  should  comprise  6-8  members  each  serving  a 
4--year  term.  The  Board  should  have  national  distribution  with  balanced 
representation  of  women  and  minorities;  the  first  requirement  is  that  each  member 
be  an  outstanding  scientist.  The  Scientific  Director,  with  the  advice  of  the 
Chairperson  of  the  Board  of  Scientific  Counselors,  will  provide  a  provisional 
list  of  potential  nominees  to  the  Institute  Director.  The  Institute  Director 
will  present  the  final  list  of  nominees  to  the  Deputy  Director  for  Intramural 
Research,  NIH,  who  will  select  and  appoint  the  Counselors.  This  process  will 
allow  appropriate  input  from  those  most  knowledgeable  about  the  specific  needs 
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of  each  Board  while  assuring  that  the  actual  selection  and  appointment  is 
independent  of  the  Institute.  The  Chairperson  of  the  Board  of  Scientific 
Counselors  will  be  selected  by  the  Scientific  Director  with  the  advice  of  the 
Institute  Director,  usually  from  one  of  the  members  serving  for  a  fourth  year. 

While  the  entire  Board  will  be  expected  to  participate  in  each  of  the  review 
sessions,  normally  there  will  be  too  few  Counselors  with  the  specific  expertise 
necessary  to  evaluate  critically  all  of  the  investigators  under  review. 
Therefore,  the  Board  will  be  supplemented  for  each  review  with  ad  hoc  reviewers 
selected  by  the  Chairperson  of  the  Board  with  the  advice  of  the  Scientific 
Director. 

IV.  REVIEW  OF  THE  INTRAMURAL  PROGRAM 

A.  GENERAL  FEATURES:  The  individual  needs  of  each  Institute  are  probably  best 
served  by  different  specific  mechanisms  of  scientific  review.  Within  certain 
specific  requirements,  therefore,  each  Institute  should  be  allowed  to  develop 
its  own  procedures.  The  Deputy  Director  for  Intramural  Research,  NIH,  will 
evaluate  the  details  of  each  Institute's  procedures  both  to  assure  that  the 
uniform  requirements  of  the  NIH  are  met  and  to  communicate  the  experiences  of 
the  different  processes  to  all  of  the  Scientific  Directors. 

Scientific  reviews  by  each  Board  of  Scientific  Counselors  supplemented  by  ad 
hoc  members  must  be  held  at  least  semi-annually.  Each  Laboratory  and  Branch 
must  be  reviewed  at  least  once  every  four  years.  At  a  minimum,  each  tenured 
investigator  of  the  Laboratory  or  Branch  must  be  reviewed.  The  ad  hoc  reviewers 
will  assure  an  impartial  scientific  review  of  each  individual  by  national  and 
international  standards  of  excellence;  the  permanent  Board  members  will  be  able 
to  place  the  specific  information  into  the  general  context  of  the  Institute's 
program  needs  and  resources. 

B.  MATERIAL  TO  BE  PROVIDED  TO  THE  REVIEWERS:  For  each  scientist  to  be  reviewed, 
the  Board  will  be  provided  beforehand  with  a  short,  written  summary  of  the 
presentation  (emphasizing  accomplishments  since  the  last  review  and  a  precis  of 
ongoing  work,  including  a  short  statement  of  future  plans),  a  curriculum  vitae, 
a  list  of  publications  of  the  last  5  years,  a  few  representative  reprints, 
enumeration  of  support  personnel  (post-doctoral  fellows,  technicians,  secretar¬ 
ies),  and  space  assignment.  The  Laboratory/  Branch  Chief  will  provide  a  summary 
of  the  administrative  organization  of  the  Laboratory/Branch  including  its 
personnel,  space  allocations,  budget  information,  and  contract  support  and 
Cooperative  Research  and  Development  Agreements. 

C.  REVIEW  PROCESS:  Generally,  the  Laboratory/Branch  Chief  will  present  an 
overview  of  the  research  activities.  The  review  of  the  individual  scientists 
may  take  the  form  of  oral  presentations  to  the  assembled  Board,  followed  by 
discussion,  or  a  mixture  of  oral  presentations  and  separate  interviews  with  one 
or  more  members  of  the  Board  and  ad  hoc  reviewers.  The  Board  will  have  Executive 
Sessions  to  discuss  the  merits  of  the  individual  presentations  as  well  as  broader 
aspects  of  the  Intramural  Program.  The  Board  may  wish  to  meet  with  the 
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Laboratory/Branch  Chiefs  to  clarify  its  understanding  of  the  scientific 
presentations  and  provide  the  Laboratory/  Branch  Chief  an  opportunity  to  respond 
to  its  questions  before  the  written  report  is  prepared.  Before  the  Board 
adjourns,  an  oral  summary  will  be  presented  to  the  Scientific  Director.  The 
Institute  Director  and  the  Deputy  Director  for  Intramural  Research,  NIH,  or  their 
designees  will  be  invited  to  these  sessions.  The  Board  will  provide  a  written 
Report  to  the  Scientific  Director  within  one  month  of  the  review. 

D.  RESPONSE  TO  THE  REVIEW:  The  relevant  portion  of  the  Board's  Report  will  be 
given  to  the  Laboratory/Branch  Chief  by  the  Scientific  Director,  and  the 
Scientific  Director  and  Laboratory/Branch  Chief  will  discuss  the  evaluations 
and  recommendations  in  the  Report.  The  Scientific  Director  may  also  wish  to 
discuss  the  Report  privately  with  one  or  more  of  the  investigators  under  review. 
Within  six  months  of  the  review,  the  Scientific  Director  will  respond  to  the 
Board  indicating  areas  of  agreement  and  disagreement  and  any  possible  changes 
that  may  be  undertaken.  Within  one  year  of  the  review,  a  written  response  will 
be  provided  to  the  Board  by  the  Scientific  Director  summarizing  the  results  of 
any  actions  taken. 

E.  DISPOSITION  OF  THE  REVIEW:  Within  six  months  of  the  review,  the  Scientific 
Director  will  transmit  the  Board's  Report  and  the  Scientific  Director's  response 
to  the  Institute  Director,  the  NIH  Deputy  Director  For  Intramural  Research,  and 
the  Director,  NIH.  The  Report  will  be  distributed  to  the  Scientific  Directors 
of  all  of  the  Institutes  and  discussed  at  a  regularly  scheduled  meeting  of  the 
NIH  Board  of  Scientific  Directors.  In  addition,  the  Scientific  Director  will 
annually  present  to  the  Institute's  National  Advisory  Council  the  written 
material  reviewed  by  the  Board  the  previous  year  and  the  Board's  Reports. 

IV.  REVIEW  OF  THE  SCIENTIFIC  DIRECTORS 


The  Board  of  Scientific  Counselors  is  in  the  best  position  to  provide  the 
Institute  Director,  who  has  the  ultimate  responsibility  for  the  quality  of  the 
Intramural  Program,  with  external  evaluations  of  the  performances  of  the 
Scientific  Director.  From  its  continued  interaction  with  the  Intramural  Program, 
the  Board  of  Scientific  Counselors  is  well  informed  on  the  goals  and  accomplish¬ 
ments  of  the  intramural  program.  Indeed,  evaluation  of  the  performance  of  the 
Scientific  Director  is  inherent  in  the  aggregate  of  the  reviews  of  the 
Laboratories  and  Branches  of  the  Intramural  Program,  and  in  the  Board's 
evaluation  of  the  response  of  the  Scientific  Director  to  its  recommendations. 
Therefore,  once  every  four  years  the  Chairperson  of  the  Board  of  Scientific 
Counselors  will  provide  a  written  report  to  the  Institute  Director  evaluating 
the  review  process,  the  overall  direction  of  the  intramural  program,  and  its 
major  accomplishments.  The  Institute  Director  may  request  an  evaluation  of  the 
Scientific  Director  by  the  Chairperson  each  year. 
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Review  and  Evaluation  of  Intramural  Research  -  1989 

John  C.  Eberhart,  Ph.D. 

Office  of  Intramural  Research 


A.  In  late  1988  the  Institute  of  Medicine  (IOM)  Committee  to  Study 
Strategies  to  Strengthen  the  Scientific  Excellence  of  the  NIH  Intramural 
Research  Program  released  its  report,  A  Healthy  NIH  Intramural  Program: 
Structural  Change  or  Administrative  Remedies?  Among  other  issues  the 
Report  commented  on  the  "peer  review"  process  by  which  boards  of 
scientific  counselors  composed  of  outside  scientists  evaluate  the 
intramural  research  programs  of  the  institutes.  The  authors  of  the 
report  write  that  "the  committee  has  four  major  recommendations  designed 
to  assure  that  the  quality  of  the  intramural  program  be  maintained  at  a 
high  level  for  the  future.  The  first  three  relate  to  assuring  that  the 
review  process  can  be  seen  to  be  rigorous  and  leads  to  wise  use  of 
intramural  resources.  .  . "  Two  of  the  three  emphasize  the  importance  of 
the  scientific  director  in  an  intramural  program,  and  recommend  both 
that  scientific  directors  should  be  paid  more  and  that  each  one  and  his 
or  her  program  "should  be  reviewed  as  a  whole  every  four  years  by  an 
external  group."  The  third  recommendation,  intended  to  "improve  the 
external  validity  of  the  review  process,"  states  "that  a  panel  chaired 
by  a  member  of  the  Director's  Advisory  Committee  (DAC)  should  be 
established  to  monitor  the  intramural  research  program  review."  The 
report  emphasizes  that  the  "monitoring"  should  "take  care  not  to 
replicate  the  activities  of  the  boards  of  scientific  counselors." 

In  response  to  these  recommendations  by  the  IOM  Committee  the  Director 
of  NIH  has  appointed  a  committee,  chaired  by  a  member  of  the  DAC  and 
including  several  former  members  of  boards  of  scientific  counselors,  to 
review  the  evaluation  process  now  in  use  in  the  NIH  intramural  programs 
and  to  make  any  recommendations  they  believe  would  improve  it.  To 
assist  the  work  of  this  committee  the  present  paper  is  intended  to  serve 
as  a  reasonably  factual  account  of  what  now  takes  place  in  the  process 
of  intramural  evaluation. 

This  report  is  thus  a  survey  and  analysis  of  intramural  evaluation  in 
1989,  and  is,  in  a  sense,  an  updating  of  the  1982  report  by  the  writer 
on  "Review  and  Evaluation  in  Intramural  Research." 

B.  "Evaluation"  in  intramural  research  and  "peer  review"  of  external  grant 
appl ications. 

One  reason--perhaps  the  principal  reason--for  the  IOM  recommendations 
about  the  review  of  intramural  research  is  the  committee's  view  that, 
regardless  of  the  dissimilarities  of  the  intramural  and  extramural 
programs,  external  review  of  both  should  be,  or  appear  to  be,  equally 
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critical,  tough,  objective,  unbiased.  This  seems  to  reflect  the  feeling 
of  some  outside  scientists  who  depend  on  NIH's  extramural  funding  for 
support  of  their  research  that  their  research  grant  applications  are 
reviewed  more  critically  than  are  intramural's  ongoing  programs. 

Whether  the  I0M  Committee  believes  that  intramural  reviews  are  actually 
less  critical  and  objective,  or  that  they  merely  appear  that  way,  is  not 
clear.  It  may  not  be  relevant,  since  a  widespread  perception  of  soft 
evaluation  for  intramural  scientists,  whether  true  or  not,  would  in  the 
long  run  be  destructive. 

Nevertheless  some  comments  are  in  order  about  NIH's  two  principal 
program  segments. 

Intramural  research  is  a  series  of  programs  operated  by  12  NIH 
institutes,  3  non-NIH  institutes  on  the  NIH  campus,  and  3  divisions  and 
centers.  The  intramural  programs  are  staffed  by  full-time  salaried 
scientists  plus  a  large  number  of  non-salaried  "guest"  scientists.  Some 
4,845  scientists  with  M.D.s  or  Ph.D.s  currently  work  in  the  intramural 
programs,  of  whom  3,843  are  paid  by  NIH  and  1,002  are  guest  researchers 
and  volunteers.  The  research  of  this  staff,  plus  the  technical  and 
support  staff,  is  funded  by  an  appropriation  (in  FY  1989)  of 
$692,272,000,  which  is  12  percent  of  the  total  NIH  appropriation  for 
research.  The  other  88  percent  supports  the  extramural  research 
program. 

The  extramural  research  program  is  a  grant-in-aid  and  contract  program 
providing  support  for  research  projects  and  programs  proposed  by 
scientists  in  universities  and  other  research  institutions.  Its 
importance  to  the  nation  is  incalculable,  since  it  procures  for  the 
long-term  effort  to  promote  the  nation's  health  the  skill  and  ingenuity 
of  accomplished  scientists  no  matter  where  they  may  work.  The 
extramural  program  embodies  a  set  of  principles  which  have  characterized 
it  since  NIH  research  grants  were  first  authorized  45  years  ago:  (a) 
the  merit  of  proposed  research  should  determine  which  applications 
receive  support;  (b)  that  merit  can  best  be  determined  by  independent 
experts  brought  in  from  outside  the  federal  government;  and  (c)  the 
relevance  of  proposed  research  to  the  missions  of  the  NIH  can  be 
considered  in  making  distinctions  between  proposals  of  apparent  equal 
merit.  Such  judgments  of  merit  must  be  made  in  order  to  decide  how  the 
money  for  research  grants  is  to  be  spent.  The  judgments  of  merit,  in 
other  words,  are  a  necessary  tool  in  the  dispensation  of  funds  from  the 
research  grants  appropriation.  An  alternative  approach--to  make  lump 
sum  grants  to  universities  and  other  research  institutions,  from  which 
the  institutions  would  decide  which  scientists'  work  should  be 
supported--has  been  rejected  in  the  past  by  extramural  scientists,  who 
seemingly  prefer  that  individual  allocations  of  research  funds  should  be 
made  by  the  NIH. 

The  outside  experts  who  judge  research  grant  applications  are  grouped 
into  "study  sections"  which  meet  two  or  three  times  a  year,  usually  in 
Bethesda,  to  review,  often,  a  hundred  or  more  applications.  The  20,000 
or  more  applications  received  annually  keep  more  than  a  hundred  study 
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sections  busy.  This  peer  review  process  is  a  large  scale  enterprise, 
costly  in  staff  and  money,  which  is  justified  only  because  its  judgments 
on  merit  are  generally  accepted  as  careful,  informed,  and  reasonably 
objective,  and  it  satisfies  most  applicants  as  to  its  fairness.  It  owes 
a  very  great  deal  to  the  hundreds  of  outside  scientists  who  are  willing 
to  work  so  hard  in  evaluating  projects. 

The  intramural  program,  to  go  back  to  it  for  a  moment,  contrasts  sharply 
in  structure  and  funding  with  the  extramural  program.  Based  on  the 
well-founded  assumption  that  an  institution  dedicated  to  research  can 
organize  and  run  a  research  program,  NIH's  several  institutes, 
responsive  to  the  expressed  will  of  the  Congress,  have  organized, 
staffed,  supervised,  and  evaluated  research  programs  that  complement 
extramural  research  in  the  mission  areas.  To  recruit  and  retain 
talented  research  staff  the  Institutes  confer  tenure  on  their  best 
scientists,  and  operate  their  programs  in  such  a  way  as  to  develop  and 
enhance  the  competence  of  staff  scientists,  and  to  enable  those  who 
perform  successfully  to  make  careers  in  the  intramural  program.  The 
Congress  annually  appropriates  money  for  each  Institute;  so  much  for 
extramural  (88  percent  of  the  research  funds),  and  the  remainder  (12 
percent)  for  the  intramural  program.  With  the  intramural  appropriation 
the  institute  director  and  scientific  director  generally  set  up  budgets 
for  individual  intramural  laboratories  and  branches,  and  in  many 
institutes  laboratory  and  branch  chiefs  then  establish  budgets  for 
sections  and/or  individual  investigators.  These  allocations  of  funds 
and  other  resources  (space  and  positions)  are  based  on  two  things:  the 
supervisors'  (scientific  directors'  and  laboratory  chiefs')  knowledge  of 
the  scientists  and  their  accomplishments  and  potential,  and  (2)  the 
evaluation  of  scientists  and  research  units  by  boards  of  scientific 
counselors. 

The  fact  that  external  review  by  boards  has  been  a  fact  of  intramural 
life  for  more  than  30  years  should  not,  however,  obscure  the  fact  that 
skillful  management  and  direction  by  program  leaders  is  the  sine  qua  non 
of  a  successful  operating  program.  A  good  program  can  put  up  with 
uncritical  outside  reviews;  no  program  can  be  good  without  skillful  and 
critical  administration.  Though  the  individual  intramural  scientists 
have  a  high  degree  of  freedom  in  the  selection  and  pursuit  of  problems, 
they  are  each  part  of  a  hierarchical  organization  headed  by  a  scientific 
director  and  a  laboratory  chief.  They  are  responsible  to  these  as  well 
as  to  their  own  research  goals  and  scientific  standards.  Such  an 
operating  research  program,  in  contrast  to  a  program  of  grants-in-aid, 
does  not  need  a  prospective  "peer  review"  system  in  order  to  function. 
Decisions  on  allocation  of  budgets  are  not  based  on  written 
applications,  as  in  a  grants  program,  but  on  close  career-long  knowledge 
of  a  cadre  of  carefully  chosen  scientists.  Though  any  judgment  is 
potentially  fallible,  and  honest  men  and  women  may  disagree  on  research 
promise  (hence  the  valuable  correction  provided  by  the  boards  of 
scientific  counselors),  there  is  no  problem  of  allocating  resources  that 
the  program  leaders  cannot  handle.  All  decisions  cannot  be  made  by 
committees,  or  boards.  If  there  is  merit  in  a  large  in-house  biomedical 
research  program,  and  the  IOM  Report  is  just  the  latest  outside  review 
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to  say  there  is,  program  leaders  must  have  authority  to  make  decisions 
that  maximize  the  long-term  effectiveness  of  the  program.  They  are 
expert,  full-time,  and  dedicated.  Others  might  differ  with  their 
judgments,  but  one  of  the  tasks  of  management  is  to  manage,  and  make 
decisions,  even  if  others  might  differ  as  to  their  wisdom.  Research 
conducted  by  industry  is  managed  (and  that  means  continual  evaluation) 
by  company  officers  without  periodic  inspection  by  outside  visitors,  and 
research  in  privately  endowed  organizations  is  often  done  the  same  way. 
People,  by  themselves,  can  run  programs.  They  can  manufacture,  mine, 
publish,  broadcast,  build  bridges,  heal  the  sick,  write  laws,  and  do 
research,  often  successfully.  The  NIH  intramural  programs  are  no  less 
able  to  do  this  than  other  private  and  governmental  organizations. 

Why,  then,  did  the  NIH,  under  the  leadership  of  Dr.  James  A.  Shannon, 
invite  outside  experts  30  years  ago  to  form  boards  of  scientific 
counselors  to  review  NIH's  intramural  research  programs  and  advise  the 
Institutes  on  high  spots  and  weak  spots,  changes  that  might  improve  the 
programs,  and  provide  counsel  to  scientific  directors  and  laboratory 
chiefs?  The  boards,  formed  in  1956,  were  not  regarded  as  analogues  of 
the  extramural  study  sections,  but  rather  as  a  way  of  using  the 
country's  best  scientific  experts  to  help  scientific  directors  guide 
their  programs  wisely.  There  was  recognition  also  that  such  a  system  of 
outside  scientific  review  would  have  value  in  public  accountability  for 
the  program,  a  not  inconsiderable  contribution. 

Finally,  how  does  evaluation  of  intramural  research  relate  to  extramural 
peer  review? 

1.  Extramural  peer  review  is  prospective  review  by  study  sections  of 
research  proposals  from  thousands  of  outside  scientists.  It  is 
designed  to  select  for  funding  the  best  of  the  proposals  that  are 
relevant  to  the  missions  of  the  institutes  concerned. 

2.  Intramural  research  evaluation  is  two  things:  (a)  a  continual 
review  of  all  intramural  research  by  scientific  directors  and 
laboratory  chiefs,  done  as  part  of  their  management  responsibility 
for  initiating,  guiding,  and  overseeing  a  research  program;  and 
(b)  a  repetitive  retrospective  review  of  each  institute's  program 
by  a  board  of  scientific  counselors,  supplemented  by  ad  hoc 
reviewers,  to  assess  programs,  critique  performance  of  research 
staff,  and  recommend  needed  changes. 

NIH  regulations  governing  review  of  intramural  research. 

Since  1985,  the  process  by  which  boards  of  scientific  counselors  review 
and  evaluate  intramural  research  has  been  governed  by  Chapter  3005  in 
the  NIH  Manual.  Until  1981,  each  Institute's  board  had  developed  its 
own  method  of  review;  in  that  year  Dr.  Robert  Goldberger,  Deputy 
Director  for  Science,  issued  a  two-page  set  of  minimum  guidelines  for 
board  operations.  The  1985  Manual  Chapter,  prepared  after  the  1982 
Eberhart  study  of  the  boards,  established  a  set  of  requirements  that  all 
boards  were  expected  to  meet.  It  was  revised  and  reissued  in  1986  after 


Congress,  in  Public  Law  99-158  (the  Health  Research  Extension  Act  of 
1985)  added  a  few  specifications.  Because  institute  intramural  programs 
and  review  boards  conform  to  these  requirements  (and  in  a  number  of 
cases  exceed  them)  it  is  worth  summarizing  what  the  Manual  Chapter  says. 

1.  NIH  Manual  3005:  Review  and  Evaluation  of  Intramural  Research  by 
Boards  of  Scientific  Counselors. 

It  is  NIH  policy,  as  prescribed  by  the  Director  and  stated  in  the 
chapter,  that  all  intramural  research  shall  be  reviewed  every  four 
years  or  oftener  by  boards  of  scientific  counselors.  To 
facilitate  such  reviews  members  of  the  boards  are  to  be  provided 
with  a  variety  of  written  material  in  advance  of  board  meetings, 
to  acquaint  board  members  with  the  laboratory  or  branch  being 
reviewed.  Included  are  to  be  research  summaries  and  future  plans 
by  the  scientists  to  be  reviewed,  plus  their  CV's  and 
bibliographies,  staff  lists,  existing  research  contracts,  and  an 
agenda  for  the  meeting.  A  copy  of  the  last  board  report  of  the 
laboratory  concerned  is  to  be  made  available  to  the  board.  The 
work  of  all  independent  scientists  in  the  laboratory  is  to  be 
reviewed  and  evaluated,  as  are  any  younger  scientists  being 
considered  for  tenure. 

The  format  of  the  meeting  at  which  the  review  is  to  be  carried  out 
is  left  to  the  Scientific  Director  of  the  Institute,  who  works  it 
out  with  his  Board.  The  chapter  states  that  "Review  formats  found 
to  be  useful  in  the  past  include;  (a)  oral  presentations  by 
scientists  to  a  plenary  session  of  the  board,  (b)  individual 
interviews  of  scientists  by  board  members  or  consultants,  and  (c) 
small  group  interviews  by  one  or  more  board  members  or  consultants 
with  members  of  a  section,  unit,  or  working  group."  Board  members 
may  visit  the  laboratories  if  they  wish. 

The  board  chairman  is  responsible  for  writing  the  report 
summarizing  and  evaluating  a  laboratory's  work.  He  is  assisted  by 
other  board  members  and  ad  hoc  consultants,  who  take 
responsibility  for  sections  of  the  report.  The  final  report  is  to 
be  sent  within  four  months  of  the  review  to  the  Scientific 
Director,  who  distributes  copies  to  the  BID  Director,  the  NIH 
Deputy  Director  for  Intramural  Research,  and  the  Director,  NIH. 

The  report  is  subsequently  reviewed  by  the  Board  of  Scientific 
Directors,  and  a  report  is  made  to  the  Institute's  national 
advisory  council  on  the  research  reviewed  and  the  results  of  the 
review.  The  impact  of  the  reviews  goes  in  two  directions:  the 
comments  and  evaluations  of  individual  scientists  are  communicated 
to  them  by  the  Scientific  Director;  and  the  latter  is  responsible 
for  making  any  necessary  changes  following  recommendations  made  in 
the  board  report.  Finally,  the  Scientific  Director  reports  back 
to  the  board,  one  or  two  meetings  later,  his  actions  in  response 
to  the  board's  recommendations.  He  is  not  required  to  implement 
every  recommendation,  but  he  must  either  do  so  or  explain  why  he 
finds  a  particular  recommendation  impracticable  or  unsuitable. 
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The  process  is  straightforward  and  in  theory  simple,  though  in 
practice  it  becomes  complicated  and  requires  a  high  level  of 
effort  on  the  part  of  many  people,  staff  and  consultants.  To 
illustrate  the  complexity,  one  Scientific  Director  has  put 
together  a  77-page  loose  leaf  notebook  to  describe  the  review 
process  as  used  in  his  intramural  program  for  the  information  of 
board  members  and  intramural  staff  alike.  Most  of  the  others  have 
not  so  far  provided  such  detailed  written  descriptions  for  their 
boards. 

D.  The  intramural  evaluation  process  in  practice  in  1989. 

Starting  from  the  core  requirements  as  summarized  above,  several 
variations  in  patterns  of  board  activities  have  emerged.  Four  will  be 
described.  The  first  is  the  modal  pattern,  which  characterizes  nine  of 
the  NIH  institutes  and  the  two  ADAMHA  institutes  whose  intramural 
programs  follow  the  NIH  system.  A  second  has  been  developed  by  one 
institute,  NIAID,  and  merits  an  individual  description.  A  third,  shared 
by  the  three  NCI  divisions  with  intramural  programs  (DCBD,  DCT,  and  DCE) 
is  different  because  reviews  are  actually  made  by  "site  visit  teams," 
chaired  by  a  member  of  the  rather  large  (15-20  member)  division  board. 
Those  three  boards,  in  addition  to  reviewing  and  approving  site  visit 
reports,  also  perform  concept  review  of  contract  proposals  and  are 
active  on  some  grant  matters.  The  fourth,  a  variation  on  the  NCI 
pattern,  has  evolved  in  NICHD.  Here  again,  site  visit  teams  make  the 
reviews,  and  the  board  deals  with  those  reviews  and  more  general  matters 
about  the  intramural  program. 

1.  The  modal  pattern. 

Most  institutes  have  separate  organizations  to  handle  intramural 
research  and  extramural  grant  and  contract  programs,  so  that  their 
boards  of  scientific  counselors  can  confine  their  efforts  to  a 
review  of  intramural  research.  Most  boards  also  want  each 
scientist  to  describe  his  own  work  and  answer  questions  about  it 
before  the  board.  Hence  the  most  frequent  pattern  of  board 
action--the  modal  pattern--embodies  those  features. 

To  characterize  this  process  it  is  useful  to  describe  a  composite 
board  in  action,  which  in  most  respects  will  be  similar  to  the 
boards  in  nine  NIH  institutes.  Each  of  the  nine  will  differ  in 
minor  ways  from  this  composite,  but  in  essence  they  are  similar 
enough  to  be  described  together.  The  descriptions  will  be 
chronological  so  that  the  full  scope  of  board  actions  can  be  seen. 

(a)  A  laboratory  or  branch  is  selected  for  review  (or  two  labs, 
depending  on  size).  Reviews  are  scheduled  several  years  in 
advance,  since  each  lab  is  to  be  reviewed  at  least  every  four 
years.  The  selected  laboratory  is  notified  to  begin  preparations 
for  the  review. 
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(b)  Dates  for  the  review  are  set  up  in  consultation  with  board 
members,  usually  at  a  preceding  meeting,  or  by  telephone.  Any  ad 
hoc  consultants  needed  to  provide  additional  expertise  for  review 
of  the  laboratory  in  question  are  selected,  by  the  scientific 
director  or  by  him  in  consultation  with  the  board  chairman,  and 
invited  to  the  meeting. 

(c)  Under  the  guidance  of  the  scientific  director,  the  lab  chief 
and  lab  scientists  prepare  the  usually  voluminous  set  of  written 
materials  which  are  sent  to  board  and  ad  hoc  members  a  month  or  so 
in  advance  of  the  board  meeting.  By  studying  these  loose  leaf 
notebooks  the  visitors  can  be  well  prepared  to  understand  and 
appraise  the  research  presented  to  them.  The  comprehensive  nature 
of  the  advance  written  material  can  be  seen  from  the  following 
list,  all  or  most  of  which  is  sent  by  each  institute. 

•  For  each  scientist  being  reviewed,  a  C V,  bibliography, 
summary  of  recent  research  and  plans  for  future  research, 
and  recent  reprints. 

•  A  staff  list  of  the  laboratory. 

•  An  organization  chart  of  the  IRP  or  program  and  the  lab. 

•  The  annual  report  of  the  laboratory  for  the  previous  year. 

•  The  last  board  review  of  that  laboratory,  probably  four 
years  earlier. 

•  The  laboratory  budget  (by  some  institutes). 

•  Drawings  showing  laboratory  space. 

•  A  list  of  intramural  research  contracts. 

•  A  meeting  agenda  and  list  of  board  members  and  ad  hoc 
consultants. 

(d)  The  meeting.  All  intramural  reviews  are  done  at  a  meeting 
of  board  and  ad  hoc  members  where  scientists  and  reviewers  meet 
face  to  face.  None  are  done  by  mail.  The  meetings  occur  twice  a 
year,  in  Bethesda  for  most  reviews,  or  in  Research  Triangle  Park, 
North  Carolina  (for  NIEHS),  Baltimore  (for  most  of  NIA),  or 
Hamilton,  Montana,  for  the  Rocky  Mountain  Laboratories  of  NIAID. 

In  general  they  last  two  days  (1-1/2  to  2-1/2),  a  period  made  up 
of  a  varied  mix  of  oral  presentations,  questions,  interviews, 
executive  sessions  for  board  discussions  and  report  drafting,  and 
often  a  final  feedback  session  for  institute  and  NIH  officials. 

Our  composite  board  begins  its  meeting  with  an  executive  session 
from  8  to  10  P.M.  on  day  1,  attended,  in  addition  to  board  and  ad 
hoc  members,  only  by  the  scientific  director,  his  deputy,  his 
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clinical  director,  and  sometimes  the  institute  director.  The 
scientific  director  briefs  the  group  on  the  general  scope  of  the 
intramural  program,  especially  changes  since  the  last  meeting,  and 
answers  questions  about  budgets,  NIH  leadership,  Congressional 
pressures,  etc.  The  review  scheduled  for  the  next  two  days  is 
brought  up  and  the  scientific  director  outlines  any  issues  or 
problems  he  wants  the  board  to  attend  to.  To  share  the  burden  of 
producing  a  report,  the  chairman  usually  assigns  board  and  ad  hoc 
members  to  the  roles  of  primary  and  secondary  reviewers.  They 
will  attend  closely  to  the  scientists  they  have  been  assigned  to 
and  will  draft  a  report--description  and  eval uation--on  each. 

At  nine  the  next  morning  the  group  convenes  for  the  first  all  day 
session.  The  opening  may  be  somewhat  ceremonial,  with  the 
institute  director  in  attendance,  and  in  some  institutes  the 
scientific  staff  of  the  laboratory  being  reviewed,  to  see  and  hear 
the  board  and  to  hear  their  colleagues  report  on  research. 
(Scientific  Directors  and  Boards  differ  in  the  importance  they 
attach  to  allowing  staff  to  be  present  as  spectators  at  the 
reporting  sessions.)  After  30  minutes  or  so  of  general  briefing, 
the  lab  chief  gives  an  overview  presentation  for  his  laboratory, 
characterizing  the  laboratory  program  and  probably  touching  on  the 
presentations  to  follow.  He  and  the  others  may  be  questioned 
during  or  after  his  talk--the  atmosphere  is  informal  and  what  the 
board  wants  in  the  way  of  information  they  get.  Following  the  lab 
chief,  the  other  independent  investigators  each  take  20  to  30 
minutes,  followed  by  10  to  15  minutes  discussion,  to  describe  the 
research  they  have  been  doing,  summarize  what  they  have 
accomplished  and  its  significance,  and  then  outline  the  directions 
they  intend  to  pursue  subsequently.  They  may  be  closely 
questioned,  and  will  be  if  their  account  is  not  clear,  if  there 
are  questions  about  the  significance  of  the  problem  or  the  methods 
used,  or  if  other  discrepancies  are  noted.  At  times  very  superior 
research  is  the  subject  of  much  discussions  too,  because  board 
members  can  be  as  interested  in  exploring  the  best  as  in 
criticizing  what  appears  flawed.  They  do  not  have  the  narrow  task 
of  putting  a  grade  on  each  scientist  and  his  research,  and  then 
going  home.  They  are  interested  in  the  whole  program  in  which 
they  are  advising,  and  their  aim  is  high  quality  for  the  long 
term,  not  just  distinguishing  between  the  sheep  and  the  goats,  if 
any.  It  is  worth  saying  that  many  times  it  is  a  pleasure  to  watch 
a  good  board  operate,  and  to  observe  the  interchanges  with  the 
scientists. 

At  noon  all  leave  but  the  board,  for  whom  a  buffet  lunch  is  likely 
to  be  brought  in  and  an  executive  sessions  begins.  The  morning's 
program  is  reviewed,  its  strengths  and  weaknesses  pointed  out,  and 
a  consensus  begins  to  develop  about  the  way  the  research  is  to  be 
characterized  in  the  report  and  what  recommendations  if  any,  are 
to  be  made.  After  an  hour  and  a  half  or  so  the  board  reconvenes 
in  reporting  session  for  an  afternoon  of  further  presentations  and 
discussion.  At  four  P.M.,  the  board  may  take  an  hour  to  visit  the 
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laboratories  and  talk  to  younger  staff  members  or  students,  before 
returning  for  a  5-6  P.M.  executive  session  to  review  the 
afternoon's  program.  Some  boards  then  have  dinner  and  spend  the 
evening  writing  drafts  of  their  observations  and  evaluations;  a 
few  boards  have  a  social  evening  and  dinner  with  the  scientific 
director  and  others  in  the  host  institute. 

The  second  day  follows  pretty  much  the  pattern  of  the  first.  If 
only  one  lab  is  being  reviewed  the  second  morning  is  probably  a 
drafting  session.  If  two  labs  are  under  study  the  second  will 
consume  most  of  the  second  day,  with  a  couple  of  executive 
sessions.  The  second  evening  will  clearly  be  an  evening  for 
writing,  in  preparation  for  a  feed-back  session  on  the  third 
morning.  At  this  session  the  members  of  the  board  read  their 
evaluations  for  the  benefit  of  the  scientific  director,  institute 
director,  deputy  director  for  intramural  research,  and  possible 
other  officials,  and  entertain  any  questions  or  other  opinions. 
There  are  few  of  the  latter. 

At  one  of  the  executive  sessions  the  scientific  director  reports 
to  the  board  on  actions  taken  as  a  result  of  the  board's 
recommendations  at  its  previous  meeting.  Sometimes  this  report  is 
in  writing,  but  more  usually  it  is  oral. 

2.  The  NIAID  Format. 

Review  of  the  NIAID  intramural  program  by  its  board  of  scientific 
counselors  differs  from  the  composite  format  just  described  in 
that  it  places  heavy  reliance  on  one-on-one  interviews  of 
individual  scientists  by  board  members.  Formal  oral  presentations 
are  generally  limited  to  those  by  lab  chiefs  and  perhaps  section 
chiefs. 

The  NIAID  board  has  8  members,  a  typical  number,  and  usually  eight 
ad  hoc  members,  where  most  boards  have  4  to  6.  Prior  to  the 
meeting  each  member  receives  a  27-page  "Information  Resource  and 
Reference  Manual"  which  summarizes  board  duties  and 
responsibilities,  and  such  program  information  as  organizational 
charts,  personnel  information,  travel  provisions,  maps,  and  a 
description  of  the  meeting  format.  In  addition,  the  member 
receives  the  usual  information  about  the  lab  to  be  reviewed-- 
research  summaries  by  the  lab  chief,  a  description  of  each  project 
in  the  laboratory,  and  information  concerning  personnel  involved, 
collaborating  units,  publications,  and  future  plans. 

Also  included  are  c.v.'s  for  each  scientist  who  is  to  be 
interviewed  and  a  meeting  agenda.  The  Scientific  Director  has 
already  assigned  primary  reviewers  for  each  scientist,  and  those 
assignments  are  tabbed  in  the  lab  book. 

The  meeting  lasts  2-1/2  days,  and  it  will  review  one  large  lab  or 
two  or  three  small  labs.  After  a  30-minute  opening  executive 
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session  the  first  morning  the  lab  chief  presents  a  summary  of  his 
laboratory  program,  a  task  some  lab  chiefs  share  with  one  or  more 
section  chiefs.  Potential  candidates  for  tenure  are  also  given 
time  to  present  their  work.  At  lunch  the  scientific  director 
states  his  questions  or  concerns  about  the  laboratory  and  lists 
the  individuals  being  considered  for  tenure  or  promotion.  He 
considers  this  a  very  important  session.  The  afternoon  is  devoted 
to  individual  interviews,  done  in  the  laboratories.  Every 
scientist  who  has  been  at  NIH  for  two  years  or  more  is  interviewed 
twice,  tenure  candidates  and  lab  chiefs  and  section  heads  three 
times.  Each  member  of  the  board  will  conduct  6  to  7  interviews, 
all  individually.  Most  last  30  minutes.  The  staff  seem  to  like 
the  interviews  and  some  comment  that  they  don't  last  long  enough. 
Interviewers  also  examine  lab  notebooks  and  any  posters  available. 

In  the  evening  the  scientific  director  entertains  the  Board,  lab 
scientists  and  institute  officials  at  his  home. 

The  second  morning  is  used  to  complete  the  interviews.  Lunch  is 
again  a  working  session,  during  which  the  scientific  director 
gives  an  oral  follow-up  on  the  previous  review  and  distributes  the 
written  response  of  the  lab  chief  reviewed.  The  scientific 
director  also  gives  a  "state  of  the  institute"  report,  and 
presents  contract  renewals  for  review. 

In  the  afternoon  final  interviews  are  completed  by  3:30  or  4:00, 
followed  by  an  executive  session  into  the  evening  to  write  up  each 
individual  scientist,  discuss  individuals,  sections,  and  the 
laboratory. 

It  is  noteworthy  that  each  board  member  writes  out  an  appraisal  of 
each  scientist  he  interviews,  and  that  these  write-ups  are  for  the 
information  of  the  Scientific  Director  and  the  scientist 
concerned,  but  not  a  part  of  the  official  report.  Any  significant 
evaluations  are,  of  course,  included  in  the  final  report. 

At  8:30  on  the  third  morning  board  members  meet  to  read  their 
draft  reports  to  each  other,  and  then  meet  privately  with  each  lab 
chief.  No  outsiders  are  present  in  this  feed  back  session  -  even 
the  scientific  director  stays  away.  The  board  summarizes  for  the 
lab  chief  the  essence  of  their  report  on  his  unit  and  gives  him 
ample  time  to  explain  or  argue  with  them.  The  lab  chiefs  take 
full  advantage  of  that  opportunity,  and  it  is  often  a  productive 
session.  After  this  the  board  welcomes  outsiders  -  the  institute 
director,  scientific  director,  deputy  director  for  intramural 
research,  and  possibly  others,  and  summarizes  their  reviews  and 
recommendations  on  each  laboratory  involved. 

All  reports  have  now  been  written  and  are  left  with  the  scientific 
director  for  preparing  final  drafts  and  return  to  board  members 
for  editing.  They  return  the  edited  drafts  within  two  weeks. 
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The  final  stage  is  reporting  back  to  the  scientists  the  comments 
by  the  board.  This  the  scientific  director  does  himself,  meeting 
individually  with  each  of  the  staff  people  reviewed,  over  a  two  to 
three  month  period,  and  going  over  with  him  or  her  the  appraisals 
written  by  board  members,  which  are  given  to  the  scientists.  The 
scientific  director  supplies  all  this  information  to  section 
chiefs  and  finally  to  lab  chiefs.  And  then,  at  the  September 
meeting  of  the  National  Advisory  Allergy  and  Infectious  Diseases 
Council,  he  presents  to  the  Council  a  status  report  on  the 
intramural  program  and  the  activities  of  the  board  of  scientific 
counselors.  The  laboratory  book  and  the  board  report  are 
available  to  the  Council. 

3.  The  NCI  System. 

The  three  intramural  research  programs  of  the  National  Cancer 
Institute  differ  from  those  of  the  rest  of  the  NIH  in  that  each  is 
organized  into  a  Division  which  includes  a  number  of  grant  and 
contract  programs.  Each  Division  has  a  board  of  scientific 
counselors  which  oversees  not  only  the  intramural  labs  and 
branches  but  also  contract  programs  and  some  grant  programs.  As  a 
consequence  of  this  organization,  as  well  as  certain  preferences 
on  the  part  of  NCI  leadership,  there  are  characteri stic 
differences  between  NCI  and  most  of  the  other  intramural  programs 
in  the  way  research  is  reviewed  and  evaluated.  The  differences 
incl ude: 

(a)  The  boards  of  scientific  counselors  are  larger,  from  15  to 
20  members,  than  are  other  boards,  and  meet  three  times  a  year, 
rather  than  twice. 

(b)  The  boards  themselves  do  not  review  laboratory  research 
programs--that  is  done  by  site  visit  teams,  each  chaired  by  a 
board  member  and  including  one  or  more  additional  board  members 
plus  6  to  10  ad  hoc  members.  Site  visit  teams  usually  review  only 
one  laboratory,  and  there  may  be  3  to  6  such  visits  a  year. 

(c)  The  site  visit  team  reports,  prepared  in  the  usual  way,  are 
presented  by  the  chairman  of  the  team  to  the  board  of  scientific 
counselors  for  discussion  and  acceptance,  modification,  or 
rejection.  After  approval  by  the  Board  the  reports  are  given  to 
the  division  director  (scientific  director),  NCI  director,  NIH 
officials,  and  the  Board  of  Scientific  Directors.  Each  laboratory 
chief  receives  a  copy  of  the  report  on  his  lab,  and  he 
communicates  selected  portions  to  the  scientists  reviewed. 

(d)  The  site  visit  team  members  are  generally  selected  by  the 
chairman  of  the  team,  who  himself  is  usually  appointed  by  the 
chairman  of  the  Board.  The  scientific  director  makes  suggestions 
on  all  this;  but  the  choices  are  not  his.  He  is  thus  somewhat 
further  removed  from  the  evaluation  process  than  are  most 
scientific  directors  outside  NCI. 
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(e)  The  NCI  intramural  review  process  is  described  in  some 
detail  in  a  9-page  printed  pamphlet,  and  the  Division  of  Cancer 
Treatment  has  developed  its  own  detailed  loose  leaf  notebook  on 
"Procedures  for  Review  of  Intramural  Research,"  with  time 
schedules,  sample  formats,  and  full  explanations  of  all  phases  of 
the  process  in  some  75  pages. 

(f)  The  NCI  division  directors  provide  more  detailed  information 
to  their  boards  and  site  visitors  about  laboratory  space  and 
budgets  than  do  most  other  scientific  directors,  at  least  in  part 
in  response  to  requests  from  board  members  and  site  visit  teams. 

(g)  The  format  of  site  visit  team  meetings  is  conventional,  with 
presentations  by  scientists,  discussions  and  questions,  and 
executive  sessions  to  discuss  findings  and  to  draft  reports. 
Laboratories  may  be  visited  by  site  visitors  if  they  wish. 

(h)  As  with  other  institutes,  board  recommendations  are 
carefully  reviewed  and  responded  to  orally  and  in  writing. 

(i)  In  general,  the  system  of  evaluation  used  by  the  NCI 
Divisions  is  somewhat  more  formal,  more  isolated  from  the 
scientific  director,  and  more  elaborately  planned  and  organized 
than  in  the  earlier  described  modal  system.  Though  the  NCI 
Scientific  Directors  like  their  present  format,  not  all  of  the 
other  scientific  directors  find  it  suitable  for  their  programs. 

4.  The  NICHD  Variation. 

The  NICHD  reviewing  process  is  a  variant  of  the  NCI  patterns  in 
that  actual  lab  evaluations  are  carried  out  by  site  visit  teams 
who  report  back  to  the  9  member  Board  of  Scientific  Counselors. 

The  special  characteristics  of  the  NICHD  approach  are  as  follows: 

(a)  The  9  member  board  meets  twice  a  year  for  one  day.  It  does 
not  review  grant  and  contract  programs.  During  the  one-day 
meeting  the  scientific  director  briefs  the  board  on  the  state  of 
the  intramural  program,  doing  what  he  calls  "reviewing  the  whole 
program"  at  both  meetings  each  year.  Usually  one  scientist  gives 
a  broad  scientific  report.  The  afternoon  is  spent  going  over 
reports  from  previous  site  visits  and  the  responses  to  them  by  lab 
chiefs  and  the  scientific  director. 

(b)  Site  visit  teams  are  chaired  by  a  member  of  the  board,  who 
selects  members  of  his  team  from  a  list  that  includes  his  own 
nominations  as  well  as  those  suggested  by  the  scientific  director. 
There  are  usually  three  site  visits  a  year,  each  reviewing  one 
lab.  A  thick  information  book  is  supplied  a  month  before  the 
meeting,  with  each  scientist  to  be  reviewed  providing  a  10-12  page 
single-spaced  account  of  his  research--  problems,  methods, 
results,  conclusions  of  past  work,  and  future  plans.  Also 
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included  are  the  usual  c.v.'s  and  bibliographies,  staff  lists,  and 
resource  information  (budgets,  space,  positions). 

(c)  The  site  visits  usually  last  2-1/2  days,  Sunday  night 
through  Tuesday.  On  Monday,  scientists  present,  usually  for  an 
hour  each,  all  day,  and  the  board  begins  to  draft  reports  in  the 
evening.  On  Tuesday  morning  small  teams  of  site  visitors  (a 
primary  and  2  secondary  reviewers)  visit  each  scientist  in  his 
laboratory.  They  will  interview  the  scientist  and  other 
laboratory  staff,  including  post-docs,  and  even  students.  Writing 
of  the  report  occupies  the  afternoon,  and  is  almost  complete  by 
adjournment  at  4  to  5  o'clock. 

(d)  The  site  visit  report  is  edited  and  finalized  in  the  usual 
way.  Within  two  months  the  lab  chief  makes  a  written  response  to 
the  report's  recommendations,  and  this  is  reviewed  by  the  Board  at 
the  same  time  they  review  the  report  itself.  A  year  later  the 
scientific  director  reports  orally  to  the  Board  what  has  happened 
to  the  lab. 

In  a  way  this  approach  combines  some  features  of  the  modal  system,  the 
NCI  method,  and  the  NIAID  format.  The  NIH  has  encouraged  institutes  to 
develop  appraisal  methods  that  they  feel  are  especially  appropriate  for 
their  own  research  programs. 

E.  Concl  usion 


A  few  concluding  remarks  are  in  order. 

1.  Review  and  evaluation  as  now  carried  out  in  the  NIH  intramural 
research  program  require  a  major  investment  of  time,  energy  and 
money  for  research  staff  and  administration.  Such  use  of  program 
resources  seems  justified. 

2.  Without  exception  the  institutes  and  their  scientific  directors 
take  seriously  the  task  of  outside  evaluation  of  their  intramural 
programs.  It  is  a  high  priority  for  all,  at  the  top  of  the  annual 
agenda.  As  one  scientific  director  recently  said,  "We  don't  want 
the  scientists  to  write  grant  applications,  but  we  do  want  them  to 
feel  that  they've  been  subjected  to  a  very  rigorous  review." 

3.  There  is  no  ready  way  to  compare  evaluations  by  the  boards  of 
scientific  counselors  with  those  of  study  sections  in  the 
extramural  program.  The  programs  are  differently  structured  and 
the  reviews  serve  different  purposes.  The  one  (extramural) 
reviews  grant  applications  to  guide  funding  decisions.  The  other 
(intramural)  reviews  research  retrospectively  (with  an  eye  also  on 
future  plans)  to  assist  the  scientific  director  in  program 
management.  It  has  important  long-term  (institutional)  aims  as 
well  as  helping  in  short  term  decisions. 

4.  The  recommendation  by  the  IOM  that  the  reviews  and  evaluations  by 
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boards  of  scientific  counselors  be  "monitored"  by  a  subcommittee 
of  the  Director's  Advisory  Committee  seems  susceptible  to  several 
interpretations.  We  have  felt  that  the  currently  envisaged 
analysis  by  a  subcommittee  of  the  Director's  Advisory  Committee 
into  the  nature  and  quality  of  the  current  procedures  for  review 
of  intramural  research  addresses  the  IOM  concerns. 
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Appendix  G 

Comparison  of  Provisions  in 
Chapter  3005,  NIH  Working  Paper 
Proposals  and  Consultant  Panel 
Recommendations 


Appendix  G 

Intramural  Review:  A  Procedural  Comparison 


Issue  NIH  Manual,  Chapter  3005 

Boards  of  Scientific  Counselors:  Boards  of  scientific  counselors 

Composition,  Selection  and  Use  nominated  by  scientific  director  (with 

the  exception  of  NCI),  approved  by  the 
institute  director,  and  the  Deputy 
Director  for  Intramural  Research,  NIH, 
and  appointed  by  the  Director,  NIH. 


Chairperson  selection,  the  number  of 
members  and  their  terms  is  not  stated. 


Ad  hoc  Reviewers 


Ad  hoc  consultants  used  as  the  scien¬ 
tific  director  deems  necessary. 
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Working  Paper 

Board  of  scientific  counselors  members 
nominated  by  the  scientific  director 
with  advice  from  the  chairperson  of 
the  board;  a  final  list  is  presented  by 
the  institute  director  and  appointed  by 
the  Deputy  Director  for  Intramural 
Research,  NIH. 


The  chairperson  is  selected  by  the 
scientific  director,  with  the  advice  of 
the  institute  director. 

Six  to  eight  members  serving  4-year 
terms. 


Panel  Report 

Nominees  selected  by  the  chairperson 
of  the  board  of  scientific  counselors 
with  the  advice  of  the  scientific  direc¬ 
tor  (with  the  exception  of  the  NCI). 

The  institute  director  and  the  Deputy 
Director  for  Intramural  Research,  NIH, 
provide  a  final  review  and  approval. 
Appointments  are  made  by  the  Direc¬ 
tor,  NIH. 

Selection  of  chairperson  not  stated. 


The  institutes  should,  to  the  extent  pos¬ 
sible,  increase  the  number  of  indi¬ 
viduals  serving  on  their  boards. 

The  term  of  office  should  be  increased 
to  5  years. 

Upon  completion  of  their  appoint¬ 
ment,  scientific  counselors  should  be 
surveyed  on  issues  of  program  quality 
and  the  scientific  review  process. 


Ad  hoc  reviewers  selected  by  the  chair-  Ad  hoc  reviewers  should  be  selected 
person,  board  of  scientific  counselors,  by  the  chairperson,  board  of  scientific 

with  advice  from  the  scientific  director,  counselors,  with  the  advice  of  the 

scientific  director. 
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Review  Format 


Issue  NIH  Manual,  Chapter  3005 

Organization  and  structure  of  meet¬ 
ings...  will  be  left  to  the  discretion  of 
the  scientific  director.  Past  formats  in¬ 
clude:  oral  presentations  to  a  plenary 
session  of  the  board  of  scientific  coun¬ 
selors,  individual  interviews  by  board 
of  scientific  counselors  members  or 
consultants,  and  small  group  inter¬ 
views.  The  reviewers  should  be  given 
the  opportunity  to  visit  the  research 
facilities,  if  they  wish. 

Evaluations  must  address,  at  a  mini¬ 
mum,  the  following  questions:  are 
good  questions  being  asked,  are  ap¬ 
proaches  appropriate,  and  are  the 
resources  available  appropriate  to  the 
accomplishments  to  date? 
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Working  Paper 

Within  certain  specific  requirements..., 
each  institute  should  be  allowed  to 
develop  its  own  procedures.  The 
Deputy  Director  for  Intramural 
Research,  NIH,  will  evaluate  the 
details...  to  assure  that  the  uniform 
requirements  of  the  NIH  are  met  and 
to  communicate  the  experiences  of  the 
different  processes  to  all  of  the  scien¬ 
tific  directors. 

Reviews  must  be  held  at  least  semi¬ 
annually.  Each  laboratory  and  branch 
must  be  reviewed  at  least  once  every  4 
years.  At  a  minimum,  each  tenured  in¬ 
vestigator...  must  be  reviewed. 


Panel  Report 

Within  certain  specific  requirements, 
each  institute  should  be  allowed  to 
develop  its  own  procedures  for  the 
organization  and  structure  of  the 
review  meetings.  The  Deputy  Director 
for  Intramural  Research,  NIH,  should 
evaluate  the  procedures  to  assure  that 
the  uniform  requirements  of  the  NIH 
are  met  and  to  communicate  the 
experiences  of  the  processes  to  all  of 
the  scientific  directors. 

The  boards  of  scientific  counselors 
should  provide  evaluation  and  advice 
on  science,  tenure  and  promotion 
decisions,  resource  allocation,  specific 
projects  including  new  areas  of 
development,  and  other  administrative 
matters. 

Uniform  standards  for  scientific 
review  procedures  should  be 
developed  and  codified. 
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Issue 

Information  Provided  Reviewers 


NIH  Manual,  Chapter  3005 

For  each  laboratory /branch  being 
reviewed:  (a)  a  list  of  all  employees  to 
include  name,  occupational  specialty, 
type  of  appointment  and  grade;  (b)  a 
list  of  all  contracts. 


For  each  scientist: 

(c)  current  Curriculum  Vitae  and 
bibliography  with  recent  relevant 
reprints;  (d)  summary  of  current 
research  and  statement  of  future  plans; 
(e)  the  amount  of  technical  and 
postdoctoral  support. 

Procedures  shall  require  that  the 
reviewing  entity  be  provided  a  written 
description  of  the  research  to  be 
reviewed.  The  Conference  Report  for 
Section  413(b)(7)  of  the  PHS  Act  notes 
that  "the  written  description  should 
present  the  overall  past  accomplish¬ 
ments  of  the  laboratory  and  its  inves¬ 
tigators  since  the  last  review  was 
conducted.  The  written  description 
might  also  present:  the  general  aims, 
objectives  and  projected  directions  of 
the  research  projects  to  be  conducted; 
appropriate  background  information 
on  possible  future  research  projects;  a 
brief  summary  of  the  general  research 
plan;  and  an  indication  of  the  impor¬ 
tance  of  the  research  projects." 

A  copy  of  the  last  board  of  scientific 
counselors  review  of  the  laboratory/ 
branch  should  be  available  to  the 
board. 
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Working  Paper 

For  each  laboratory/branch  reviewed: 
a  summary  of  the  administrative 
organization...  including  personnel, 
space  allocations,  budget  information, 
contract  support,  and  Cooperative 
Research  and  Development  Agreements. 


For  each  scientist: 
a  short  written  summary  of  ac¬ 
complishments  since  the  last  review,  a 
precis  of  ongoing  work,  a  short  state¬ 
ment  of  future  plans,  a  Curriculum 
Vitae,  a  bibliography  of  the  last  5  years, 
a  few  representative  reprints,  enumera¬ 
tion  of  support  personnel,  and  space 
assignment. 


Panel  Report 

For  each  laboratory/branch  reviewed: 
all  board  members  should  be  provided 
with  a  summary  of  the  organization  of 
the  laboratory  to  be  reviewed,  the 
various  sections,  all  personnel,  space 
usage,  operating  budget,  and  outside 
contracts  and  Cooperative  Research 
and  Development  Agreements. 

Each  scientist  being  reviewed: 
should  supply  details  of  ongoing 
work,  a  curriculum  vitae,  reprints  of 
the  3  to  5  most  important  recent 
publications,  and  an  analysis  of  the 
amount  of  support  staff  and  space  that 
the  scientist  uses. 

Scientific  counselors  should  be 
provided  with  orientation  guidelines 
that  describe  the  review  procedures, 
the  goals  of  the  process,  and  the 
responsibilities  of  the  counselors. 

These  guidelines  should  be  published 
by  the  Deputy  Director  for  Intramural 
Research,  NIH. 
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Issue  NIH  Manual,  Chapter  3005 

Board  Report  and  Follow-up  The  report  of  the  board  of  scientific 

counselors  is  to  be  a  narrative  critique 
from  the  chairperson  of  the  board,  fol¬ 
lowing  the  outline  preferred  by  the 
scientific  director. 

Within  4  months,  the  board  of  scien¬ 
tific  counselors  report  is  sent  to  the 
scientific  director  who  transmits  it  to 
the  institute  director,  the  Deputy  Direc¬ 
tor  for  Intramural  Research,  NIH,  and 
the  Director,  NIH.  The  Office  of  Intra¬ 
mural  Affairs  sends  to  NIH  Board  of 
Scientific  Directors  and  schedules 
discussion. 


The  scientific  director  reports  back  to 
the  board  of  scientific  counselors  on 
actions  taken  or  explains  why  it  was 
found  impossible  or  inadvisable  to  fol¬ 
low  the  recommendations  of  the  board. 
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Working  Paper 

Before  the  board  adjourns,  an  oral  sum¬ 
mary  will  be  presented  to  the  scientific 
director,  with  the  institute  director  and 
the  Deputy  Director  for  Intramural 
Research,  N1H,  invited  to  attend. 

The  board  of  scientific  counselors  will 
provide  a  written  report  to  the  scien¬ 
tific  director  within  1  month  of  the 
review.  Relevant  portions  of  the 
report  are  shared  by  the  scientific  direc¬ 
tor  with  the  laboratory /branch  chiefs 
for  discussion. 


Within  6  months  of  the  review,  the 
scientific  director  responds  to  the 
board  of  scientific  counselors  indicat¬ 
ing  areas  of  agreement  and  disagree¬ 
ment  and  possible  changes  that  may 
occur,  transmits  the  report  and  his 
response  to  the  institute  director,  the 
Deputy  Director  for  Intramural 
Research,  NIH,  and  the  Director,  NIH. 
The  report  is  distributed  to  the  Board 
of  Scientific  Directors,  and  discussed  at 
regularly  scheduled  meetings. 


Panel  Report 

Before  the  board  adjourns,  an  oral  sum¬ 
mary  of  the  review  should  be  pre¬ 
sented  to  the  scientific  director,  the 
institute  director  and  the  Deputy  Direc¬ 
tor  for  Intramural  Research,  NIH. 

Within  a  month  of  adjournment,  the 
board  should  provide  a  written  report 
of  its  review  to  the  scientific  director. 
Those  portions  of  the  report  that  per¬ 
tain  to  particular  laboratory/branch 
chiefs  should  be  provided  to  them  by 
the  scientific  director.  The  scientific 
director  and  the  laboratory/branch 
chief  should  discuss  the  evaluations 
and  recommendations  in  the  report. 

Within  6  months  of  receiving  the 
board's  report,  but  prior  to  the  next 
meeting  of  the  board,  the  scientific 
director  should  respond  to  the  board 
indicating  areas  of  agreement  and  dis¬ 
agreement  and  any  possible  changes 
that  may  be  undertaken.  A  copy  of  the 
report  and  the  response  of  the  scien¬ 
tific  director  should  be  sent  to  the 
director  of  the  institute,  the  Deputy 
Director  for  Intramural  Research,  NIH, 
the  Director,  NIH,  and  the  institute's 
national  advisory  council. 
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Issue  NIH  Manual,  Chapter  3005 

Board  Report  and  Follow-up  (conFd) 


The  institute  annually  transmits  the 
board  of  scientific  counselors  reports 
and  the  results  of  its  reviews  to  its 
national  advisory  council. 


Review  of  Scientific  Directors  "Every  independent  intramural 

scientist...  must  be  reviewed  and 
evaluated."  The  scientific  director  is 
not  specifically  mentioned. 
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Working  Paper 

After  1  year  the  scientific  director 
provides  the  Board  of  Scientific  Direc¬ 
tors  with  a  summary  of  results  of  any 
actions  taken. 

The  scientific  director  annually 
presents  the  institute  advisory  council 
the  written  material  reviewed  by  the 
board  the  previous  year  and  the 
board's  reports. 


[E]  valuation  of  the  performance  of  the 
scientific  director  is  inherent  in  the 
aggregate  of  the  reviews  of  the 
laboratories  and  branches...  Once 
every  4  years,  the  board  of  scientific 
counselors  chair  provides  to  the  insti¬ 
tute  director  a  report  evaluating  the 
review  process,  the  program's  overall 
direction,  and  its  major  accomplish¬ 
ments.  The  institute  director  may 
request  an  evaluation  of  the  scientific 
director  by  the  chairperson  each  year. 


Panel  Report 

No  later  than  1  year  after  the  review,  a 
written  response  should  be  provided 
to  the  board  of  scientific  counselors  by 
the  scientific  director  summarizing  the 
results  of  any  actions  taken. 

The  institute's  national  advisory  coun¬ 
cil  should  be  provided  an  overview  of 
the  institute's  intramural  research 
program  at  least  once  a  year.  The  over¬ 
view  should  include,  at  a  minimum,  a 
discussion  of  the  board's  report  and 
the  response  of  the  scientific  director. 


The  scientific  and  administrative  per¬ 
formance  of  the  scientific  directors 
should  be  evaluated  every  4  years  by 
the  boards  of  scientific  counselors. 

The  chairperson  of  the  board  of  scien¬ 
tific  counselors  should  provide  the 
institute  director  with  a  written  evalua¬ 
tion  of  the  scientific  director,  including 
a  discussion  of  the  overall  direction  of 
the  intramural  program,  and  the 
efficacy  of  the  review  process.  The 
institute  director  may  request  an 
evaluation  of  the  scientific  director  by 
the  chairperson  each  year. 


